e-Petri dishes, devices, and systems having a light detector for sampling a sequence of sub-pixel shifted projection images

ABSTRACT

An e-Petri dish comprising a transparent layer having a specimen surface and a light detector configured to sample a sequence of sub-pixel shifted projection images of a specimen located on the specimen surface. The sub-pixel shifted projection images associated with light from a plurality of illumination angles provided by an illumination source.

CROSS-REFERENCES TO RELATED APPLICATIONS

This non-provisional application is a continuation-in part applicationof U.S. patent application Ser. No. 13/281,287 entitled “ScanningProjective Lensless Microscope System,” filed on Oct. 25, 2011, which isa non-provisional application of, and claims priority to, U.S.Provisional Patent Application No. 61/406,916 entitled “ScanningProjective Microscopy System for 2D and 3D Imaging,” filed on Oct. 26,2010, and U.S. Provisional Patent Application No. 61/482,531 entitled“ePetri: An On-Chip Cell Imaging Platform based on Sub-Pixel PerspectiveSweeping Microscopy” filed on May 4, 2011. This non-provisionalapplication is also a non-provisional application of, and claimspriority to, U.S. Provisional Patent Application No. 61/482,531 entitled“ePetri: An On-Chip Cell Imaging Platform based on Sub-Pixel PerspectiveSweeping Microscopy” filed on May 4, 2011 and U.S. Provisional PatentApplication No. 61/448,964 entitled “Electronic Petridish with BrightField and Fluorescence Imaging Capabilities for Cell Culture Monitoring”filed on Mar. 3, 2011. All of those applications are hereby incorporatedby reference in their entirety for all purposes.

This non-provisional application is related to the following co-pendingand commonly-assigned patent applications, which are hereby incorporatedby reference in their entirety for all purposes:

-   -   U.S. patent application Ser. No. 13/069,651 entitled “Super        Resolution Optofluidic Microscopes for 2D and 3D Imaging” filed        on Mar. 23, 2011.

The following non-provisional application is being filed on the same dayand is hereby incorporated by reference in its entirety for allpurposes: U.S. patent application Ser. No. 13/411,103 entitled “LightGuided Pixel,” filed on Mar. 2, 2012.

STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSOREDRESEARCH OR DEVELOPMENT

This invention was made with government support under Grant No. AI096226awarded by the National Institutes of Health. The government has certainrights in the invention.

BACKGROUND OF THE INVENTION

Embodiments of the present invention generally relate to highresolution, wide field-of-view microscopes and other imaging devices.More specifically, certain embodiments relate to e-Petri dishes(electronic petridishes), e-Petri devices, and e-Petri systems for highresolution, wide field-of-view imaging.

The miniaturization of biomedical imaging tools has the potential tochange vastly methods of medical diagnoses and scientific research. Morespecifically, compact, low-cost microscopes could significantly extendaffordable healthcare diagnostics and provide a means for examining andautomatically characterizing a large number of cells, as discussed inPsaltis, D., et al., “Developing optofluidic technology through thefusion of microfluidics and optics,” Nature, Vol. 442, pp. 381-386(2006), which is hereby incorporated by reference in its entirety forall purposes. For example, miniaturized imaging systems may provide auseful alternative to large microscopes in biology labs, allowing forparallel imaging of large number of samples. Some examples of compactmicroscopes can be found in Cui, X., et al., “Lensless high-resolutionon-chip optofluidic microscopes for Caenorhabditis elegans and cellimaging,” Proceedings of the National Academy of Sciences, 105(31), p.10670 (2008), Seo, S., et al., “Lensfree holographic imaging for on-chipcytometry and diagnostics,” Lab on a Chip, 2009. 9(6), pp. 777-787,Breslauer, D., et al., “Mobile phone based clinical microscopy forglobal health applications,” (2009), Zheng, G., et al., “Sub-pixelresolving optofluidic microscope for on-chip cell imaging,” Lab on aChip, 2010. 10(22), pp. 3125-3129, which are hereby incorporated byreference in their entirety for all purposes. Conventional opticalmicroscopes have bulky optics, and have proven to be expensive anddifficult to miniaturize.

Rapid advances and commercialization efforts in complementary metaloxide semiconductor (CMOS) imaging sensor technology has led to broadavailability of cheap, high-pixel-density imaging sensor chips. In thepast few years, these imaging sensor chips enabled the development ofnew microscopy implementations that are significantly more compact andless expensive than conventional microscopy designs with bulky optics.The optofluidic microscope and the digital in-line holographicmicroscope are two examples of these new developments. Some examples ofoptofluidic microscope technologies can be found in Heng, X., et al.,“Optofluidic microscopy—method for implementing a high resolutionoptical microscope on a chip,” Lab Chip, Vol. 6, pp. 1274-1276, Cui,Xiquan, et al., “Lensless high-resolution on-chip optofluidicmicroscopes for Caenorhabditis elegans and cell imaging,” Proceedings ofthe National Academy of Science, Vol. 105, p. 10670 (2008), and Zheng,G., Lee, S A., Yang, S., Yang, C., “Sub-pixel resolving optofluidicmicroscope for on-chip cell imaging. Lab Chip,” Lab Chip, Vol. 10, pp.3125-3129 (2010) (“Zheng”), which are hereby incorporated by referencein their entirety for all purposes. Some examples of digital in-lineholographic microscopy can be found in Repetto, L., Plano, E.,Pontiggia, C., “Lensless digital holographic microscope withlight-emitting diode illumination,” Opt. Lett., Vol. 29, pp. 1132-1134(2004), (“Repetto”), Mudanyali, O., et al., “Compact, light-weight andcost-effective microscope based on lensless incoherent holography fortelemedicine applications,” Lab on a Chip, Vol. 10, pp. 1417-1428 (2010)(“Mudanyali”), Xu, W., Jericho, M., Meinertzhagen, I., Kreuzer, H.,“Digital in-line holography for biological applications,” Proc Natl AcadSci USA, Vol. 98, pp. 11301-11305 (2001) (“Xu”), Garcia-Sucerquia, J.,et al., “Digital in-line holographic microscopy,” Appl. Opt., Vol. 45,pp. 836-850 (2006) (“Garcia-Sucerquia”), Malek M., Allano, D.,Coëtmellec, S., Lebrun, D., “Digital in-line holography: Influence ofthe shadow density on particle field extraction,” Opt. Express, Vol. 12,pp. 2270-2279 (2004) (“Malek”), Isikman, S. O., et al., “Lens freeoptical tomographic microscope with a large imaging volume on a chip,”Proc Natl Acad Sci USA, Vol. 108, pp. 7296-7301 (2011), which are herebyincorporated by reference in their entirety for all purposes.

Both optofluidic and in-line holographic microscopy technologies aredesigned to operate without lenses and, therefore, circumvent theiroptical limitations, such as aberrations and chromaticity. Bothtechnologies are suitable for imaging dispersible samples, such asblood, fluid cell cultures, and other suspensions of cells or organisms.However, neither can work well with confluent cell cultures or anysample in which cells are contiguously connected over a sizable lengthscale.

In the case of an optofluidic microscope device, imaging requiresfluidic (e.g., microfluidic) flow of specimens across a scanning area.Adherent, confluent, or contiguously arranged specimens are usuallyincompatible with imaging in a fluidic mode. In addition, the field ofview may be limited by the geometry of the fluid channel.

In digital in-line holographic microscopy, the interference intensitydistribution of a target under controlled light illumination is measuredand then an image reconstruction algorithm is applied to rendermicroscopy images of the target. Two examples of algorithms can be foundin Liu, G., Scott, P., “Phase retrieval and twin-image elimination forin-line Fresnel holograms,” J Opt Soc Am A, Vol. 4, pp. 159-165 (1987)(“Liu”), Fienup, J R., “Reconstruction of an object from the modulus ofits Fourier transform,” Opt Lett, Vol. 3, pp. 27-29 (1978) (“Fienup”),Koren, G., Polack, F., Joyeux, D., “Iterative algorithms for twin-imageelimination in in-line holography using finite-support constraints, JOpt Soc Am A, Vol. 10, pp. 423-433 (1993), which are hereby incorporatedby reference in their entirety for all purposes. The image qualitydepends critically on the extent of the target, the scattering propertyand the signal-to-noise ratio (SNR) of the measurement processes, whichare described in Mudanyali, and Garcia-Sucerquia, Malek, Fienup, andalso in Lai, S., King, B., Neifeld, M A, “Wave front reconstruction bymeans of phase-shifting digital in-line holography,” Opt Commun., Vol.173, pp. 155-160 (2000) (“Lai”), and Rodenburg, J., Hurst, A., Cullis,A., “Transmission microscopy without lenses for objects of unlimitedsize,” Ultramicroscopy, Vol. 107, pp. 227-231 (2007) (“Rodenburg”),which are hereby incorporated by reference in their entirety for allpurposes. The method works well for well-isolated targets, such asdiluted blood smear slides. However, such approaches appear to have notbeen applied to targets that occupy more than 0.1 mm2 in totalcontiguous area coverage with submicron resolution, as found in Repetto,Madanyali, Xu, Garcia-Sucerquia, and also in Biener, G., et al.,“Combined reflection and transmission microscope for telemedicineapplications in field settings,” Lab Chip, Vol. 11, pp. 2738-2743(2011), which is hereby incorporated by reference in its entirety forall purposes.

The reason for this limitation is well-known: the loss of phaseinformation during the intensity recording process. In order to recoverthe phase information, object support has to be used in the iterativephase recovery algorithm, which involves the light field propagationback and forth between the imaging domain (where the intensity data areapplied) and object domain (where a priori object constrains areapplied), as discussed in Liu. When the test object is real ornonnegative, it is easy to apply the powerful normegativity supportconstraint to extract the phase information from the recordeddiffraction intensity, as discussed in Liu. However, for digital in-lineholography, light field in the object domain is complex valued and,therefore, the phase recovery is possible only if the support of theobject is sufficiently isolated (i.e., sparsity constrains) or the edgesare sharply defined (true boundary), as discussed in Rodenburg andFienup and also in Denis, L., Lorenz, D., Thiébaut, E., Fournier, C.,Trede, D., “Inline hologram reconstruction with sparsity constraints,”Opt Lett, Vol. 34, pp. 3475-3477 (2009), Zhang, F., Pedrini, G., Osten,W., “Phase retrieval of arbitrary complex-valued fields throughaperture-plane modulation,” Phys Rev A, Vol. 75, p. 043805 (2007), whichare hereby incorporated by reference in their entirety for all purposes.Furthermore, the interference nature of the technique implies thatcoherence-based noise sources, such as speckles and cross-interference,would be present and would need to be addressed, as discussed inGarcia-Sucerquia and Malek, and also in Xu, L., Miao, J., Asundi, A.,“Properties of digital holography based on in-line configuration,” OptEng, Vol. 39, pp. 3214-3219 (2000), which is hereby incorporated byreference in its entirety for all purposes. Methods for mitigatingproblems in digital in-line holographic microscopy have been reported inLai, Rodenburg and Micó, V., García, J., Zalevsky, Z., Javidi, B.,“Phase-Shifting Gabor Holographic Microscopy,” J Disp Technol, Vol. 6,pp. 484-489 (2010), which is hereby incorporated by reference in itsentirety for all purposes. The generated images based on thesemitigating methods have artifacts that may arise from interference, andare identifiably different and of lower quality than images acquiredwith conventional microscopes due to coherence based noise sources.

Embodiments of the invention are directed to systems that areimprovements over conventional optofluidic and in-line holographicsystems that use bulky optics.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the present invention relate to e-Petri dishes, e-Petridevices, and e-Petri systems for high resolution, wide field-of-viewimaging. The e-Petri dish includes a specimen surface and well forreceiving and containing a specimen being imaged. The e-Petri dish alsoincludes a light detector such as a CMOS imaging sensor chip with asensing surface. The e-Petri dish also includes a transparent layerbetween the sensing surface and the specimen surface. The transparentlayer may be part of the light detector. The e-Petri device includes thee-Petri dish and a structure for receiving the e-Petri dish. The e-Petridevice also includes an illumination source (e.g., smartphone) forproviding illumination from different illumination angles at differenttimes to a specimen located on the specimen surface. The illuminationfrom the different illumination angles generates projections of thespecimen on the sensing surface. The light detector captures a sequenceof sub-pixel shifted projection images of the specimen at the sensingsurface. The e-Petri device also includes a processor as part of thelight detector or separate from the light detector. The processor canreconstruct a higher resolution (HR) image from the sequence ofsub-pixel shifted, lower resolution (LR) projection images captured bythe light detector.

One embodiment is directed to an e-Petri dish comprising a transparentlayer having a specimen surface and a light detector configured tosample a sequence of sub-pixel shifted projection images of a specimenlocated on the specimen surface. The sub-pixel shifted projection imagesare associated with light from a plurality of illumination anglesprovided by an illumination source.

Another embodiment is directed to an e-Petri device comprising atransparent layer having a specimen surface, an illumination sourceconfigured to provide light from a plurality of illumination angles to aspecimen located on the specimen surface, and a light detectorconfigured to sample a sequence of sub-pixel shifted projection imagesof the specimen. The sequence of sub-pixel shifted projection imagescorresponds to the plurality of illumination angles.

Another embodiment is directed to an e-Petri system comprising one ormore e-Petri devices and a processor. Each e-Petri devices comprises atransparent layer having a specimen surface, an illumination sourceconfigured to provide light from a plurality of illumination angles to aspecimen located on the specimen surface, and a light detectorconfigured to sample a sequence of sub-pixel shifted projection imagesof the specimen. The sequence of sub-pixel shifted projection imagescorresponds to the plurality of illumination angles. The processor isconfigured to generate a sub-pixel resolution image of the specimenbased on the sequence of sub-pixel shifted projection images from atleast one of the one or more e-Petri devices.

Another embodiment is directed to a method of automatedly generating asub-pixel resolution image of a specimen using an e-Petri system atintervals during an experiment. The method comprises introducing thespecimen on a specimen surface of an e-Petri dish. During each interval,the method provides light from a plurality of illumination angles to thespecimen. During each interval, the method also uses a light detector tocapture a sequence of sub-pixel shifted projection images correspondingto the plurality of illumination angles. During each interval, themethod also constructs a sub-pixel resolution image of the specimenbased on the sequence of sub-pixel shifted projection images and amotion vector.

These and other embodiments of the invention are described in furtherdetail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of components of an SPLM system, accordingto embodiments of the invention.

FIG. 2 is a drawing of a perspective view of components and partialcomponents of an SPLM device or an e-Petri device, according toembodiments of the invention.

FIGS. 3(a), 3(b), and 3(c) are drawings of perspective views ofcomponents and partial components of an SPLM device or an e-Petri deviceduring an illumination cycle of an imaging run, according to embodimentsof the invention.

FIG. 4(a) and FIG. 4(b) are diagrams illustrating a scanning pattern onan illuminating display, according to embodiments of the invention.

FIG. 5(a) is an LR projection image captured by a light detector of anSPLM system or an e-Petri system at a single sampling time, according toembodiments of the invention.

FIG. 5(b) is an HR image reconstructed by the SPLM system or an e-Petrisystem, according to embodiments of the invention.

FIG. 6(a) is an LR projection image of a portion of a HeLa cell specimencaptured by a light detector of an SPLM system or an e-Petri system at asingle sampling time, according to embodiments of the invention.

FIG. 6(b) is an HR image reconstructed by the SPLM system or an e-Petrisystem, according to embodiments of the invention.

FIG. 7(a) is a large field of view color HR image of a confluent HeLacell specimen constructed by an SPLM system or an e-Petri system,according to embodiments of the invention.

FIG. 7(b 1) is an LR projection image from a small region of FIG. 7(a),captured by the light detector of an SPLM system or an e-Petri system,according to embodiments of the invention.

FIG. 7(c 1) is an LR projection image from a small region of FIG. 7(b1), captured by the light detector of an SPLM system or an e-Petrisystem, according to embodiments of the invention.

FIG. 7(b 2) is a reconstructed HR image from the same small region ofFIG. 7(a) constructed by an SPLM system or an e-Petri system, accordingto embodiments of the invention.

FIG. 7(c 2) is a reconstructed HR image from a small region of FIG. 7(b2) constructed by an SPLM system or an e-Petri system, according toembodiments of the invention.

FIG. 7(d) is a conventional microscopy image of similar cells using amicroscope with 40×, NA=0.66 objective lens.

FIG. 8(a) is an HR image of a specimen having 500 nm microspheres(Polysciences) as constructed by an SPLM system or an e-Petri system,according to embodiments of the invention.

FIG. 8(b) is an HR image of a magnified small feature of the stainedHeLa cell specimen of FIG. 7 as constructed by an SPLM system or ane-Petri system, according to embodiments of the invention.

FIG. 9 is a flow chart of an exemplary operation of an SPLM device or ane-Petri device, according to embodiments of the invention.

FIG. 10 is a schematic drawing of three projections on a light detectorfrom three different illumination angles, θ₁, θ₂, and θ₃, according toan embodiment of the invention.

FIG. 11 is a schematic diagram of an e-Petri system having n e-Petridevices and an expanded view of one of the e-Petri devices, according toembodiments of the invention.

FIG. 12 is a photographic image of an e-Petri system having a singlee-Petri device, according to embodiments of the invention.

FIG. 13(a) is a photographic image of an e-Petri dish according to anembodiment of the invention, and a quarter for size comparison.

FIG. 13(b) is a photographic image of a partially disassembled e-Petridevice having the e-Petri dish of FIG. 13(a), according to an embodimentof the invention.

FIG. 13(c) is a photographic of the assembled e-Petri device of FIG.13(b), according to an embodiment of the invention.

FIG. 14 (al), FIG. 14(a 2), and FIG. 14(a 3) are conventional microscopyimages with red, green and blue LED illuminations (20× objective, 0.5N.A.).

FIG. 14 (a 4) is the color image constructed based on the red, green,and blue images in FIG. 14 (a 1), FIG. 14(a 2), and FIG. 14(a 3).

FIG. 14 (b 1), FIG. 14(b 2), and FIG. 14(b 3) are reconstructedsub-pixel resolution images of a portion of HeLa cell sample as acquiredby an e-Petri system under red, green, and blue light source scanningrespectively, according to an embodiment of the invention.

FIG. 14 (b 4) is the reconstructed sub-pixel resolution color imagebased on the red, green, and blue images in FIG. 14 (b 1), FIG. 14(b 2),and FIG. 14(b 3), according to an embodiment of the invention.

FIG. 15(a) are time-lapse reconstructed images of a portion of the HeLacell sample from a specific sub-location, as acquired by an e-Petrisystem at starting at times, t=10 hr, t=17.5 hr, t=25 hr and t=32.5 hrduring the time period of the study, according to an embodiment of theinvention.

FIG. 15(b) are graphs of the tracking trajectories of three cellfamilies annotated by a biologist and the lineage trees for these cellfamilies that were processed by the second processor of the hostcomputer of an e-Petri system, according to an embodiment of theinvention.

FIG. 16 is a flow chart of an exemplary method of automatedlyreconstructing and displaying the sub-pixel resolution images and/orother data at intervals during an experiment by an e-Petri system,according to embodiments of the invention.

FIG. 17 is a block diagram of subsystems that may be present in the SPLMsystem or an e-Petri system, according to embodiments of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The need for a high-resolution, wide-field-of-view, cost-effectivemicroscopy solution for autonomously imaging growing and confluentspecimens (e.g., cell cultures), especially in longitudinal studies, isa strong one, as discussed in Schroeder, T., “Long-term single-cellimaging of mammalian stem cells,” Nature Methods 8, pp. S30-S35 (2011),which is hereby incorporated by reference in its entirety for allpurposes. Some examples of experiments that would benefit from such asolution include: 1) the determination of daughter fates before thedivision of neural progenitor cells, as discussed in Cohen, A. R.,Gomes, F. L. A. F., Roysam, B. & Cayouette, M., “Computationalprediction of neural progenitor cell fates,” (2010); 2) the existence ofhaemogenic endothelium, as discussed in Eilken, H. M., Nishikawa, S. I.& Schroeder, T., “Continuous single-cell imaging of blood generationfrom haemogenic endothelium,” Nature 457, pp. 896-900 (2009), 3) neuraland hematopoietic stem and progenitor divisional patterns and lineagechoice, as discussed in Costa, M. R. et al., “Continuous live imaging ofadult neural stem cell division and lineage progression in vitro,”Development 138, 1057 (2011) and in Dykstra, B. et al., (National AcadSciences); 4) the in-vitro tissue culture studies using the neutral reddye, as discussed in Repetto, G., del Peso, A. & Zurita, J. L., “Neutralred uptake assay for the estimation of cell viability/cytotoxicity,”Nature Protocols 3, 1125-1131 (2008); 5) detection of toxic compound, asdiscussed in Borenfreund, E. & Puerner, J. A., “Toxicity determined invitro by morphological alterations and neutral red absorption,”Toxicology Letters 24, pp. 119-124 (1985); and 6) drug screening, asdiscussed in Cavanaugh, P. F. et al., “A semi-automated neutral redbased chemosensitivity assay for drug screening,” Investigational newdrugs 8, pp. 347-354 (1990). These cited references are herebyincorporated by reference in their entirety for all purposes. The laborintensive nature of the experiments and the challenge of efficientlyimaging large assays have plagued the experiment formats in theseexamples and others. Embodiments of the invention may address these andother challenges.

Embodiments of the invention include e-Petri dish and SPLM devices andsystems that provide a chip-scale, lensless microscopy methodology thatcan automadedly image growing and confluent specimens (e.g. cellcultures) over a wide field-of-view with sub-pixel (e.g., subcellular)resolution. In one embodiment, for example, an e-Petri device can imagea specimen with an area of 6 mm×4 mm at 660 nm resolution. These e-Petriand SPLM devices and systems can be automated to periodically orotherwise repeatedly image specimens over time. These devices andsystems can also be used image color stained specimens. Since e-Petriand SPLM devices may be compact, multiple devices can be placed in anincubator to image and track specimen changes such as cell culturegrowth directly within the incubator.

With these capabilities and others, this self-imaging microscopymethodology may significantly improve petri-based experiments (e.g.,cell cultures) and other real-time imaging procedures in medicine andscientific disciplines. For example, by using this compact, low-costmicroscopy imaging methodology, petri-dish based experiments may betransformed from the traditionally labor-intensive process to anautomated and streamlined process. Further, this methodology may cutdown on lab equipment usage and contamination risks. This possibletechnological shift from an inert petri-dish to a self-imagingpetri-dish may be timely as well, because, the cost of high performanceCMOS imaging sensors (which are widely used in cellphone cameras andwebcams) has recently reached a price point where they can be used asrecyclable or disposable components. With these capabilities, thee-Petri and SPLM devices and systems provide a smart petri-dish platformthat may be well suited for long-term (cell culture) imaging andtracking applications. For example, the e-Petri and SPLM devices andsystems may be used to explore time-resolved information in studyingsystems biology, cell growth, and in-vitro drug screening, where thecounting and tracking individual cells in an in-situ and parallel manneris difficult with conventional methods such as bulky microscopes orplate readers.

These embodiments of the e-Petri and SPLM devices and systems and otherswill be described below with reference to the accompanying drawings.Embodiments include an e-Petri system, which includes one or moree-Petri devices having an e-Petri dish with a specimen surface and wellfor receiving and containing a specimen being imaged. The e-Petri systemmay also include an incubator, external processor, and/or othercomponents according to embodiments of the invention. The e-Petri dishalso includes a light detector (e.g., CMOS imaging sensor chip) having asensing surface and a transparent layer between the sensing surface andthe specimen surface. The e-Petri device includes an e-Petri dish alongwith an illumination source (e.g., smartphone) for providingillumination from different illumination angles at different times to aspecimen located on the specimen surface of the e-Petri dish. Theillumination from the different illumination angles generatesprojections of the specimen on the sensing surface. The light detectorcaptures a sequence of sub-pixel shifted projection images at thesensing surface of the specimen. The e-Petri system also includes aprocessor, which may or may not be incorporated with any component ofthe system. The processor can reconstruct a higher resolution image fromthe sequence of sub-pixel shifted projection images captured by thelight detector. Other embodiments include an SPLM device having aspecimen surface for receiving a specimen being imaged, an illuminationsource, a light detector having a sensing surface, a transparent layerbetween the sensing surface and the specimen surface, and a processor.

The e-Petri and SPLM devices and systems use the same general imagingmethod. First, a specimen is located on the specimen surface of thedevice. During an illumination cycle, an illumination source provideslight from different illumination angles to the specimen. For example,if an LCD is used, the light element may be in the form of differentsets of light pixels that illuminate at different times to change theposition of the light element. The positions of the different sets oflight emitting component can be in a pattern designed to generatesub-pixel shifted projections of the specimen at the sensing surface ofthe light detector. The light detector can capture a sequence of thesesub-pixel shifted projection images (frames) at the sensing surface. Aprocessor can calculate the motion vector of the sub-pixel shiftedprojections using a suitable estimation algorithm. The processor canthen use a super resolution algorithm to construct a sub-pixelresolution or other HR image of the specimen from the sequence ofsub-pixel shifted projection images and the calculated motion vector.Under different imaging schemes, the e-Petri and SPLM devices/systemscan generate HR monochromatic 2D images, HR monochromatic 3D images, HRcolor 2D images, and/or HR color 3D images. In a digital focusingscheme, these devices/systems can focus a HR image of the specimen at aplane through the specimen using a motion vector calculated at thatplane to construct an HR image.

This imaging method leverages the recent broad and cheap availability ofhigh performance image sensor chips to provide a low-cost and automatedmicroscopy solution. Unlike the two major classes of lensless microscopymethods, optofluidic microscopy and digital in-line holographymicroscopy, this imaging method is fully capable of working with cellcultures or any samples in which cells may be contiguously connected.

Embodiments of the invention provide one or more technical advantages.In embodiments, the illumination source is located at a larger distancefrom the sensing surface as compared with the distance between thespecimen and the sensing surface. Thus, small translations of the lightelement correspond to larger translations of the projections on thesensing surface. With this geometry, the illumination source can easilyand accurately control and maintain sub-pixel shifts of the projections.These devices can then generate an accurate sub-pixel resolution imagebased on the accurately controlled sub-pixel shifted projections. Insome embodiments, the devices can generate images with a resolutioncompatible to conventional 20×-40× objective microscopes. This canprovide an advantage over other systems, such as prior microscanningsystems, that used actuators and controller to control sub-pixelmovements of the object or a platform holding the object.

Other advantages of embodiments are that this method may provide anautonomous, cost effective, high resolution microscopic solution forhigh resolution imaging of confluent specimens and other specimens inwhich objects in the specimen may be contiguously connected over asizable length. One advantage is that the systems and devices may bemade compact (e.g., on the chip-scale) and at lower cost than conventionmicroscopy. Since these systems and devices may be compact and low cost,multiple devices can easily be used in parallel to image many specimens.For example, several e-Petri devices may be placed in a single incubatorand monitored using the display on single laptop computer. With thiscompact platform, lab testing may become more streamlined and requireless lab equipment allocation. Another advantage is that the systems anddevices can automatedly image and produce other data, which may reducethe human labor involved in lab experiments. Some benefits of automatedimaging can be found in Levin-Reisman, I., et al., “Automated imagingwith ScanLag reveals previously undetectable bacterial growthphenotypes,” Nat Meth, 7(9), pp. 737-739 (2010), which is herebyincorporated by reference in its entirety for all purposes. Due to theautomated nature of the imaging used by the devices of embodiments,contamination risks may be reduced. Also, experimental procedures may besignificantly streamlined and may be more accurate.

Components of the SPLM systems and devices are discussed in detail inSection I, which also describes some components of e-Petri systems anddevices. The imaging method used by the SPLM and e-Petri systems anddevices is discussed in Section II. Components of the e-Petri systemsand devices are discussed in detail in Section III. Some components ofe-Petri systems and devices are also described in detail in Section I.

I. Scanning Projective Lensless Microscope (SPLM) System

FIG. 1 is a schematic diagram of components and partial components of anSPLM system 10, according to embodiments of the invention. The SPLMsystem 10 includes an SPLM device 100 and a host computer 200.

The SPLM device 100 includes a specimen surface 140 for receiving aspecimen (e.g., confluent sample). The SPLM system 10 can image at leasta portion of the specimen 150. In the illustrated example, a specimen150 with five objects 152 (e.g., cells) is located on the specimensurface 140. Although five objects 152 are shown, the specimen 150 mayhave any suitable number (e.g., 1, 2, 10, 100, 1000, etc.) or portion(s)of objects 152.

The SPLM device 100 also includes a scanning illumination source 110having a first processor 112, a first computer readable medium (CRM)114, and an illuminating display 116 (e.g., an LCD, a light emittingdiode (LED) display, etc.). The first processor 112 is in electroniccommunication with the illuminating display 116 and with the first CRM114. The illuminating display 116 includes a light element 117 (e.g.,one or more pixels of an LCD or LED display)) capable of generatingillumination 118 (e.g., incoherent light). The illuminating display 116also includes a display surface 119. The light element 117 is located atthe display surface 119 in the illustrated example. In otherembodiments, a transparent layer may be located between the displaysurface 119 and the light element 117. Also, a transparent layer may belocated outside the display surface 119 in some embodiments. Thescanning illumination source 110 also includes an x-axis, a y-axis (notshown), and a z-axis. The x-axis and y-axis lie in a plane at thedisplay surface 119. The z-axis is orthogonal to this plane.

The scanning illumination source 110 can scan (sweep) or otherwisetranslate the light element 117 to different scanning locations acrossthe display surface 119 in order to provide illumination 118 to thespecimen 150 from different illumination angles. The shifting lightelement 117 (source) of the illumination 118 generates shiftingprojections 170 (as shown in FIG. 3) of a specimen 150 on the sensingsurface 162. In FIG. 1, the light element 117 is shown at a scanningposition at a time, t during the scanning cycle of an imaging run. Eachscanning cycle of an SPLM device 100 can refer to a time interval duringwhich the scanning illumination source 110 scans or otherwise translatesthe light element 117 to the scanning locations in that particularscanning cycle. An imaging run of an SPLM device 100 can refer a timeinterval during which one or more operations of the SPLM system 10generates an HR image based on light data collected during one or morescanning cycles. In embodiments, the light element 117 may shift duringa scanning cycle to n×m scanning locations in a two-dimensional (n×m)array of scanning locations: (x_(i=1 to n), y_(j=1 to m)) on the displaysurface 119.

The SPLM device 100 also includes a light detector 160 for capturingprojection images. The light detector 160 includes a sensing surface 162having a sensing area 164. The sensing surface 162 is located at adistance, d, from the display surface 119. The light detector 160 alsoincludes a transparent layer 165 (e.g., thin transparent passivationlayer) located between the specimen surface 140 and the sensing surface162. During a scanning cycle, the illumination 118 from the lightelement 117 generates projections 170 (shown in FIG. 3) of the specimen150 on the sensing surface 162. The light detector 160 can sample(capture) one or more sequences of sub-pixel shifted LR projectionimages of the specimen 150 during a scanning cycle. Each sub-pixelshifted LR projection image can refer to an LR projection image that hasshifted a sub-pixel distance from a neighboring LR projection image inthe sequence. Neighboring LR projection images in a sequence can referto two LR projection images that are proximal in distance. In somecases, neighboring LR projection images may also be projection imagesthat have been sequentially captured in time during a scanning cycle.

As shown by a dotted line, the light detector 160 may optionally be inelectronic communication with the first processor 112 forsynchronization of sampling by the light detector 160 with scanning bythe scanning illumination source 110. The light detector 160 alsoincludes an x′-axis, a y′-axis (not shown), a z′-axis. The x′-axis andy′-axis lie in a plane at the sensing surface 162 of the light detector160. The z′-axis is orthogonal to this plane.

The SPLM system 10 also includes a host computer 200 having a secondprocessor 210, a second CRM 220 in electronic communication with thesecond processor 210, and an image display 230 in electroniccommunication with the second processor 210. The second processor 210can receive data associated with one or more sequences of sub-pixelshifted LR projection images from the light detector 150. The secondprocessor 210 can also determine a motion vector of the projections 170at the sensing surface 162 based on the data. The second processor 210can then use a suitable super resolution algorithm (SR algorithm) togenerate one or more HR (e.g., sub-pixel resolution) images of thespecimen 150 based on the motion vector and data of one or moresequences of sub-pixel shifted LR projection images. The secondprocessor 210 is in electronic communication with the second processor210 to display the HR images and/or other images.

In an exemplary imaging run of the SPLM system 10 of FIG. 1, thescanning illumination source 110 scans or otherwise translates the lightelement 117 to a two-dimensional (n×m) array of n×m scanning positionshaving the coordinates (x_(i=1 to n), y_(j=1 to m)) on the displaysurface 119. The scanning illumination source 110 scans (sweeps) orotherwise translates the light element 117 to scanning positionsaccording to a scanning pattern. Illumination 118 from the light element117 at the different scanning locations generates shifted projections170 of the specimen 150 on the sensing surface 162 of the light detector160. During scanning, the light detector 160 captures one or moresequences of sub-pixel shifted LR projection images at the sensing area164. The second processor 210 receives data for at least one of thesequences from the light detector 160. The second processor 210 candetermine a motion vector of the sub-pixel shifted projections 170 atthe sensing surface 162 from the data. The second processor 210 can alsoconstruct one or more HR images of the specimen 150 using a suitablesuper-resolution algorithm with the data from at least one of thesequences of sub-pixel shifted LR projection images of the specimen 150and/or the determined motion vector.

FIG. 2 is a drawing of a perspective view of components and partialcomponents of an SPLM device 100 or an e-Petri device, according toembodiments of the invention. The SPLM device 100 or an e-Petri deviceincludes a scanning illumination source 110 in the form of a mobilecommunication device (e.g., cell phone, tablet, etc.) and a lightdetector 160 in the form of a two-dimensional array of light detectingelements 166 (e.g., CMOS imaging sensor). The light detector 160 has asensing surface 162 and a thin transparent layer 165. A specimen 150comprising a single object 152 (e.g., cell) is located on the specimensurface 140 (not shown). The thin transparent layer 165 lies between thesensing surface 162 and the specimen surface 140. In this example,scanning illumination device 110 includes an illuminating display 116(not shown) in the form of an LCD. The LCD includes a two-dimensionalarray of light emitting components (e.g., pixels). The scanned lightelement 117 is in the form of subsequent sets of light emittingcomponents on the LCD 116 providing illumination 118 according to ascanning pattern. Each set may include one or more light emittingcomponents. The subsequent sets of light emitting components provideillumination 118 at the two-dimensional (n×m) array of n×m scanninglocations at (x_(i=1 to n), y_(j=1 to m)) on the display surface 119. InFIG. 2, the light element 117 is shown at a single scanning location inthe scanning cycle. The illumination 118 from the light element 117generates a single projection 170 of the object 152 on the sensingsurface 162 of the light detector 160.

FIGS. 3(a), 3(b), and 3(c) are drawings of perspective views ofcomponents and partial components of an SPLM device 100 or an e-Petridevice during a scanning cycle of an imaging run, according toembodiments of the invention. The SPLM device 100 or an e-Petri deviceincludes a scanning illumination source 110 providing illumination 118and a light detector 160 in the form of a two-dimensional array of lightdetecting elements 166. The light detector 160 has a sensing surface 162and a thin transparent layer 162. A specimen 150 comprising a singleobject 152 (e.g., cell) is located on the specimen surface 140. The thintransparent layer 165 lies between the sensing surface 162 and thespecimen surface 140. The SPLM device 10 or an e-Petri device alsoincludes an x-axis, a y-axis, and a z-axis. The x-axis and y-axis lie inthe plane at the sensing surface 162 of the light detector 160. Thez-axis is orthogonal to this plane. The light element (not shown) 117provides illumination 118 and generates light spots on the lightdetector 160.

In FIGS. 3(a), 3(b), and 3(c), the light element 117 (not shown) islocated at three scanning positions along the x′-axis at times: t=t_(a),t_(b), and t _(c) (a>b>c), respectively. Illumination 118 is shown fromthree different scanning position to generate three shifted projections170(a), 170(b), and 170(c) on the sensing surface 162, respectively.With certain shifts of the light element 117, the object's projection(shadow) can be shifted at sub-pixel (i.e. smaller than pixel size)increments across the light detecting elements 166 (e.g. sensor pixels)of the light detector array. At times: t=t₁, t₂, and t₃, the lightdetector 160 captures a sequence of three LR projection imagescorresponding to the three projections 170(a), 170(b), and 170(c),respectively. Any suitable number of sub-pixel shifted projections mayhave been captured at scanning times between times, t_(a) and t_(b) orbetween times, t_(b) and t_(c). A motion vector of the projections170(a), 170(b), and 170(c) at the sensing surface 162 can be determinedbased on the data from a sequence of sub-pixel shifted LR projectionimages. An HR image of the object 152 can be constructed using asuitable SR algorithm and based on the data from a sequence of sub-pixelshifted LR projection images captured by the light detector 160.

Any suitable specimen 150 may be imaged by the SPLM system 10 or theSPLM device 100. In most cases, the specimen 150 is stationary during ascanning cycle. An example of a suitable specimen 150 is a confluentsample (e.g., confluent cell cultures) having one or more objects 152(e.g., cells). Another example of a suitable specimen 150 is a sample inwhich the objects 152 are contiguously connected. The specimen 150 beingimaged may include any suitable type(s) of object(s) 150 and may includeany suitable number (e.g., 1, 10, 100, 1000, etc.) of objects 150 orportion(s) of an object 150. Suitable types of objects 150 can bebiological or inorganic entities. Examples of biological entitiesinclude whole cells, cell components, microorganisms such as bacteria orviruses, cell components such as proteins, etc. Inorganic entities mayalso be imaged by embodiments of the invention.

As used herein, a scanning illumination source 110 can refer to anysuitable device or combination of devices capable of scanning orotherwise translating a light element 117 to n scanning positions togenerate sub-pixel shifted projections 170 of a specimen 150 beingimaged at a sensing surface 162 of a light detector 160. Any number, n,of scanning positions can be used (n=1, 2, 3, 4, 5, 10, 20, 100 etc.).By moving the light element 117, the scanning illumination source 110changes the illumination angles of the illumination 118 provided to thespecimen 150. In embodiments, the scanning illumination source 110 movesthe light element 117 to scanning locations that generate a small rangeof illumination angles (e.g., +/−2 degrees) in X/Y around the normal tothe sensing surface or other plane of interest.

An example of a suitable scanning illumination device 110 is a mobilecommunication device (e.g., cell phone, tablet, etc.). Suitable scanningillumination sources 110 commercially available. Illustrated examples ofa suitable scanning illumination device 110 in the form of a smartphoneare shown in FIGS. 2 and 4. Another example of a suitable scanningillumination device 110 may be a tomographic phase microscope that usesa spatial light modulator to scan illumination.

In embodiments, the scanning illumination source 110 may include anilluminating display 116 for scanning the light element 117 to generatesub-pixel shifted projections 170 at the sensing surface 162. Anilluminating display 116 can refer to any suitable display capable oftranslating a light element 117 to scanning locations across at least aportion of a display surface 119. Suitable illuminating displays 116 arecommercially available. Some examples of suitable illuminating displays116 include monochromatic, color, or gray-scale LCDs, LED displays(e.g., display panels), television screens, LCD matrixes, etc. In theseembodiments, the illuminating display 116 may include a two-dimensionalarray of light emitting components (e.g., pixels). The array of lightemitting components may have any suitable dimension (e.g., 1000×1000,1000×4000, 3000×5000 etc.). The display surface 119 can refer to asurface of the illuminating display 116 that provides illumination 118.For example, the scanning illumination source 110 may be in the form ofa smartphone with an illuminating display 116 in the form of an LCDscreen, as shown in FIGS. 2 and 4. In other embodiments, the scanningillumination source 110 may include another device or combination ofdevices capable of scanning the light element 117 to generate sub-pixelshifted projections 170 at the sensing surface 162.

The scanning illumination source 110 may be held at a fixed positionrelative to the light detector 160 and the transparent layer 165 duringscanning in some embodiments. In these embodiments, the SPLM 100 mayinclude a suitable structure (e.g., platform, frame, etc.) or structuresto hold the scanning illumination source 110 and light detector 160 in afixed position. In some cases, such as the illustrated example of FIG.1, the scanning illumination source 110 may be held such that thedisplay surface 119 is kept approximately parallel to the sensingsurface 162 of the light detector 160 and at a distance, d, from thesensing surface 162 during scanning. In these cases, the illuminatingdisplay 116 may provide illumination 118 at angles normal to the displaysurface 119. In other cases, the scanning illumination source 110 may beheld so that the display surface 119 may be tilted at an angle fromnormal. At this angle, projections 170 from more extreme illuminationangles to be captured, leading to a more complete 3D reconstruction insome cases. In one embodiment, the scanning illumination source 110 mayinclude actuator(s) and controller(s) or other mechanism to repositionthe illuminating display 116 (e.g., LCD array) at an angle from normal.

A light element 117 can refer to a suitable device capable of providingillumination 118. The properties of the illumination 118 generated bythe light element 117 can have any suitable values. Some properties ofthe illumination 118 include intensity, wavelength, frequency,polarization, phase, spin angular momentum and other light propertiesassociated with the illumination 118 generated by the light element 117.In embodiments, the illumination 118 is incoherent light. The componentor components of the light element 117 can change over time. Forexample, in the case of an LCD, different pixels may illuminate atdifferent times to change the position of the light element 117.

In embodiments with an illuminating display 116 in the form of atwo-dimensional array of light emitting components (e.g., pixels), alight element 117 at a particular scanning time, t may be a set of asuitable number (e.g., 1, 5, 10, 100, etc.) of illuminated lightemitting components (e.g., LCD lit/pixel) in the two-dimensional array(e.g., LCD array). Each light emitting component may have a scanninglocation denoted as (x_(i), y_(j)) where i=1 . . . N; and j=1 . . . N.The light element 117 may be the illuminated pixels in the array at ascanning time in the scanning cycle. The scanning location of a lightelement 117 can refer to the coordinates of the center of the set ofilluminated light emitting components in this case. In theseembodiments, sequentially illuminated sets of light emittingcomponents(s) on an illuminating display 116 can generate light elements117 at different scanning locations during a scanning cycle.

The properties (e.g., size, properties of the illumination 118, shape,etc.) of the light element 117 may have any suitable value. Inembodiments, one or more properties of the light element 117 may vary atdifferent scanning locations in a scanning cycle. In other embodiments,the properties of the light element 117 may be constant during thescanning cycle. Some examples of suitable shapes of a light element 117are a rectangle, circle, spot, bar, etc. In embodiments with anilluminating display 116 in the form of a two-dimensional array of lightemitting components, the properties of the light element 117 can bevaried by varying the number of light emitting components in the set oflight emitting components (e.g., pixels) forming the light element 117.For example, the intensity of the illumination 118 generated by thelight element 117 can be varied by changing its number of light emittingcomponents (e.g., pixels). In embodiments, one or more properties of theillumination 118 generated by the light element 117 may change atdifferent scanning locations.

In embodiments, the intensity of the illumination 118 generated by thelight element 117 may be controlled by varying the size of the lightelement 117. In one embodiment, the size of the light element 117 mayvary at different scanning locations to generate light at approximatelythe same intensity at a single point at the plane of the sensing surface162. In this embodiment, the size, S of the light element 117 at ascanning location can be proportional to the distance, L, from thescanning location to a suitable location such as: a) the center of thearray of scanning locations, or b) the center of an illuminating display116 such as the center of an LCD on a smartphone. For example, the size,S of the light element 117 at a scanning location may be defined as:S=S_(center)×(1+L), where S_(center) is the size of the light element117 at the center of the array of scanning locations. In this way, thelight intensity received at a location at the sensing surface 162 normalto the center of the scanning locations on the display surface 119 canbe kept about constant in some cases. As another example, the size S ofthe light element 117 at a scanning location in a scanning cycle may bedefined as: S=S_(A)×(1+A), where S_(A) is the size of the light element117 at a location A of an illuminating display 116, A is the distancefrom the scanning location to the location A.

In one embodiment, the light element 117 can provide illumination 118 ofn different wavelengths λ₁, . . . , λ_(n) at different scanning timesduring a scanning cycle. The illumination 118 may be sequentially cycledthrough a series of different wavelengths as the light element 117 movesthrough scanning locations in a scanning cycle in some examples. In oneexample, the light element 117 can provide RGB illumination of threewavelengths λ₁, λ₂, and λ₃ corresponding to red, green, blue colors,respectively. The light element 117 may provide illumination 118 of thethree wavelengths λ₁, λ₂, and λ₃ sequentially during scanning times of ascanning cycle. In one case, at a scanning time t₁ illumination 118 mayhave a wavelength of λ₁, at t₂ illumination 118 may have an wavelengthof λ₂, at t₃ illumination 118 may have a wavelength of λ₃, at t₄illumination 118 may have a wavelength of λ₁, at t₅ illumination 118 mayhave a wavelength of λ₂, etc.

A scanning location can refer to the center of the light element 117.Any suitable number (e.g., 1, 100, 1000, etc.) of scanning locations maybe used in a scanning cycle. As a group, the scanning locations in ascanning cycle may cover any suitable area. In embodiments with adisplay surface 119, the scanning locations may cover the entire displaysurface 119 or may cover a portion of the display surface 119.

To shift projections 170 of the specimen 152 at the sensing surface 162,the scanning illumination source 110 can translate the light element 117to different scanning locations generating different illuminationangles. To generate a sequence of sub-pixel shifted projections 170 ofthe specimen 152 at the sensing surface 162 in some embodiments, thescanning illumination source 110 may move the light element 117 to aplurality of scanning locations designed to generate sub-pixel shiftedprojections 170. In this case, neighboring scanning locations in theplurality of scanning locations correspond to a sub-pixel shift ofneighboring projections images 170 in the sequence of projection images.Neighboring scanning locations can refer to two scanning locations thatare proximal in distance. In some cases, neighboring scanning locationsmay also be locations that are sequential in time having sequentialscanning times during a scanning cycle.

The scanning locations may form any suitable arrangement (e.g., array,circle, square, triangle, etc.). In embodiments, the scanning locationsmay be in the form of an array (e.g., one-dimensional array,two-dimensional array, or combination of one-dimensional andtwo-dimensional arrays) of scanning locations. In these embodiments, thearray of scanning locations may have any suitable dimension (e.g. 1×100,1×10, 100×100, 3000×20, 400×300 etc.). For example, the scanninglocations may be arranged in a two-dimensional (n×m) array of n×mscanning locations at (x_(i=1 to n), y_(j=1 to m)).

In embodiments with an illuminating display 116 (e.g., LCD display) inthe form of a two-dimensional array of light emitting components (e.g.pixels), the scanning locations of the light element 117 can refer tosubsequently illuminated light emitting components in thetwo-dimensional array. In these embodiments, the scanning locations ofthe light element 117 may be located at the display surface 119. Forexample, the scanning locations may be in the form of a two-dimensional(n×m) array of n×m scanning locations at (x_(i=1 to n), y_(j=1 to m)) onthe display surface 119.

In embodiments, the scanning illumination source 110 the light element117 during a scanning cycle according to a scanning pattern. A scanningpattern can refer to a description of the scanning locations (i.e.locations of the light element 117) at different times during a scanningcycle and properties (e.g., size, shape, etc.) of the light element 117at each scanning location in the scanning cycle. For example, a scanningpattern may include a two-dimensional array of scanning locations and adescription that the light element 117 moves through each rowsequentially at a constant rate. In another example, the scanningpattern may include a two-dimensional array of scanning locations and adescription that the element moves through each column sequentially at aconstant rate. As another example, the scanning pattern may include atwo-dimensional array of scanning locations and a description that theelement moves through the array randomly. The scanning pattern may alsoinclude the amount of sub-pixel shift desired between subsequent LRprojection images. The scanning pattern may also include the totalnumber of LR projection images and/or HR images desired. The scanningpattern may be stored as code on the first CRM 114 or the second CRM220. In embodiments with a scanning illumination source 110 in the formof a smartphone such as in FIG. 4, the scanning pattern may be anapplication (App) stored in the memory of the smartphone.

In embodiments such as the illustrated example of FIG. 1, the SPLMdevice 100 also includes a transparent layer 165 located between thespecimen surface 140 and the sensing surface 162. The transparent layer165 can separate the specimen 150 from the light sensitive region of thelight detector 160. The transparent layer 165 may be made of anysuitable material such as Polydimethylsiloxane (PDMS). The transparentlayer 165 may have any suitable thickness (e.g., thickness in the rangeof several hundred nanometers to microns). In some cases, thetransparent layer 165 may be a layer placed on the light detector 160.For example, the transparent layer 165 may be a passivation layer coatedor deposited on top of an imaging sensor chip. In other cases, thetransparent layer 165 may be separate from the light detector 160. Inother embodiments, the SPLM device 100 does not have a transparent layer165 and the sensing surface 162 is coincident with the specimen surface140. The transparent layer may also be comprised of multiple layers. Forexample, the transparent layer may be comprised of a passivation layerproximal the light detector 160 and a protective coating.

The distance between neighboring projections 170 is proportional to thethickness of the transparent layer 165 and the tilt/shift extent of thelight element 117. The tilt/shift extent of the light element 117 canrefer to the distance or illumination angle change between neighboringscanning locations. In some embodiments, the distance betweenneighboring scanning locations in a plurality of the scanning locationsof a scanning cycle can be designed to generate sub-pixel shiftedprojections 170. In these cases, the distance between the neighboringscanning locations can be determined based on the thickness of thetransparent layer 165 and the required incremental sub-pixel shiftsbetween neighboring projections 170.

In embodiments, the distance between neighboring scanning locations in aplurality of scanning locations may be determined to generate sub-pixelsshifts between neighboring projections 170 in a sequence of projectionimages. In these embodiments, the determined distance betweenneighboring scanning locations in the plurality of scanning locationsdirectly corresponds to a sub-pixel shift of a projection 170 at thesensing surface 162. In these embodiments, the plurality of scanninglocations directly corresponds to a sequence of sub-pixel shiftedprojection images.

In embodiments, the distance between neighboring scanning locations maybe a suitable value. In some cases, the distance between neighboringscanning locations in a given scanning cycle may be constant. In othercases, it may vary.

A scanning rate can refer to the rate of shifting between sequentialscanning locations in a scanning cycle per unit in time. A sampling ratecan refer to a rate of projection images (frames) captured by the lightdetector 160 per unit in time such as frames per second. Thesampling/scanning rate may be constant in some embodiments and may varyin other embodiments. In embodiments, the scanning rate and samplingrate are synchronized.

FIG. 4(a) and FIG. 4(b) are diagrams illustrating a scanning pattern onan illuminating display 116, according to embodiments of the invention.In this example, the scanning illumination source 110 is in the form ofa smartphone and the illuminating display 116 is in the form of an LCDscreen of the smartphone. The LCD screen includes a two-dimensionalarray of pixels of a 640×640 pixel size. During scanning, the smartphonemay be located at a suitable distance, d above the light detector 160(e.g., image sensor chip). The display surface 119 of the illuminatingdisplay 116 and the sensing surface 162 of the light detector 160 may bekept approximately parallel. The smartphone may be located so that thecenter of the display surface 119 of the illuminating display 116 isabove the sensing area 164 of the sensing surface 162 of the lightdetector 160. The illuminating display 116 includes an x-axis and ay-axis. The x-axis and y-axis lie in the plane at the display surface119 of the illuminating display 116.

FIG. 4(a) shows a light element 117 comprising a set of about 640 pixelsin the form of a bright circular spot of about 1 cm in diameter on theilluminating display 116. The light element 117 is shown at a scanninglocation at a scanning time during a scanning cycle. The light element117 may be located at the display surface 119 of the illuminatingdisplay 116.

In FIG. 4(b), the diagram of the scanning pattern includes a 15×15 arrayof scanning locations (steps) of the light element 117 during thescanning cycle. The scanning locations are shown at locations along thex-axis and y-axis in the plane of the display surface 119 of theilluminating display 116. In the illustrated example, the scanningpattern includes 15 scanning locations in the x-direction and 15scanning locations in the y-direction. In this example, the lightdetector 160 may capture 225 LR projection images based on the 225scanning locations in the scanning patter. The array of scanningpositions may be centrally located within the illuminating display 116.The arrows in FIG. 4(b) designate the order of the scanning locationsduring the scanning cycle. In this case, the light element 117 movessequentially through each row of the two-dimensional array of scanninglocations in the scanning pattern. If the light element 117 remains aconstant size as it moves away from the center of the display surface119, the intensity readout from the light detector 160 (e.g., imagesensor chip) will decrease because of the large incident angle. Tomaintain a more constant intensity readout, the size of the lightelement 117 (e.g., bright spot size) can be linearly increased as itmoves away from the center of the illuminating display 116 (e.g.,smartphone screen) in one embodiment.

Returning to FIG. 1, the scanning illumination source 110 includes afirst processor 112 in electronic communication with the illuminatingdisplay 116 and a first CRM 114 in communication with the firstprocessor 112. The first processor 112 (e.g., microprocessor) canexecute code stored on the first CRM 114 (e.g., memory) to perform someof the functions of the scanning illumination source 110. For example,the first processor 112 may execute code with a scanning pattern storedon the first CRM 114. The CRM 114 may include, for example, code with ascanning pattern, other code for scanning a light element 117, and othercodes for other functions of the scanning illumination source 110. Thefirst CRM 114 may also include code for performing any of the signalprocessing or other software-related functions that may be created bythose of ordinary skill in the art. The code may be in any suitableprogramming language including C, C++, Pascal, etc.

In embodiments, the light detector 160 may be in electroniccommunication with the first processor 112 of the scanning illuminationsource 110 to synchronize sampling of the light detector 160 with thelight element 117 being located at a scanning position. In theseembodiments, the sampling rate of the light detector 160 may besynchronized with the scanning rate of the scanning illumination source110 to capture at least one projection image 170 at each scanninglocation. In one embodiment, an electronic start sampling signal may besent to the light detector 160 from scanning illumination source 110 tocapture an LR projection image when the light element 117 is at ascanning location.

The SPLM device 100 also includes a light detector 160 (e.g., CMOSimaging sensor). As used herein, a light detector 160 can refer to anysuitable device or combination of devices capable of capturingprojection images 170 and generating one or more signals with dataassociated with the projection images 160 captured and other dataassociated with imaging. The signals with data may be in the form of anelectrical current from the photoelectric effect.

The light detector 160 includes a sensing surface 162. As used herein, asensing surface 162 can refer to the active sensing layer of the lightdetector 160. The sensing surface 162 includes a sensing area 164. Thesensing area 164 refers to a suitable area of the sensing surface 162that actively captures projections 170 during a scanning cycle. In somecases, the entire area of a sensing surface 162 is the sensing area 164.In embodiments, the specimen 150 being imaged may be located in an areaof the specimen surface 140 proximal the sensing area 162. The lightdetector 160 also includes a local x′ axis and y′ axis at a plane of thesensing surface 162.

In embodiments, the light detector 160 includes discrete light detectingelements 166 (e.g., pixels) in the form of a two-dimensional array oflight detecting elements 166, as shown in FIGS. 2 and 3. The lightdetecting elements 166 may be located on or within a surface layer ofthe light detector 160 at the sensing surface 162. Although thetwo-dimensional array of light detecting elements 166 is oriented sothat the x′-axis is parallel to the x-axis of the illuminating display116 as shown in FIGS. 2 and 3, the two-dimensional array may be orientedat any suitable angle in other embodiments.

Any suitable light detector 160 can be used. Some examples of suitablelight detectors 160 having two-dimensional arrays of light detectingelements 166 include a charge coupled device (CCD) array, a CMOS imagingsensor array, an avalanche photo-diode (APD) array, a photo-diode (PD)array, and a photomultiplier tubes (PMT) array. These light detectors160 and others are commercially available. Also, the light detector 160can be a monochromatic detector or a color detector (e.g., RGBdetector).

The light detecting elements 166 may be of any suitable size (e.g., 1-10microns) and any suitable shape (e.g., circular, rectangular, square,etc.). For example, a CMOS or CCD light detecting element 166 may be1-10 microns and an APD or PMT light detecting element 166 may be aslarge as 1-4 mm.

Due to the scattering angle of light 118 passing through a specimen 150being imaged, projection image quality can be degraded if the specimen150 is located away from the sensing surface 162 of the light detector160. In embodiments, the light detector 160 does not have a color filterand microlens layer in order to decrease the acceptance angle of eachlight detecting element and the distance between the object 152 and thesensing surface 120 (i.e. the active sensing layer). If the lightdetector 160 (e.g., a CMOS imaging sensor chip) was prefabricated with acolor filter and a microlens layer, these components may be removed todecrease the acceptance angle of each pixel and the distance between theobject 152 and the surface layer.

In embodiments, the transparent layer 165 may be placed, duringfabrication, on the light detector 160. Semiconductor and/ormicro/nanofabrication procedures may be used to place the transparentlayer 165 on the light detector 160. In some cases, the transparentlayer 165 may be placed on the light detector 160 after the color filterand microlens layer have been removed. In one case, the color filter andmicrolens layer may be removed by treating the pre-fabricated imagingsensor under oxygen plasma for a period of time (e.g., 10 minutes at 80W). The transparent layer 165 may be placed onto the imaging sensorafter the removal of the color filter and microlens layer or may beplaced on a light detector with the layer. In one case, the transparentlayer 165 may be prepared by mixing 1:10 with base and curing agent,then spin coated on a 3 in. silicon wafer followed by baking at 80degrees C.

Light data can refer to any suitable information related to the one ormore projections 170 captured by the light detecting elements 166 of thelight detector 160. For example, light data may include informationabout the properties of the projection light received such as theintensity(ies) of the light, the wavelength(s) of the light, thefrequency or frequencies of the light, the polarization(s) of the light,the phase(s) of the light, the spin angular momentum(s) of the light,and/or other light properties associated with the light received by thelight detecting element 166. Light data may also include the location ofthe receiving light detecting element(s) 166, the time that the lightwas received (sampling time or scanning time), or other informationrelated to the projection 170 received. In embodiments, each lightdetecting element 166 can generate a signal with light data based onlight associated with the projection 170 and received by the lightdetecting element 166.

An LR projection image (frame) can refer to a snapshot image sampled(captured) by the light detector 160 at a sampling time occurring duringa scanning cycle. In embodiments, the light detector 160 captures an LRprojection image at each scanning time. Each LR projection image sampledby the light detector 160 can be used to display a 2D, LR projectionimage. In embodiments with a color light detector 160, the LR projectionimage may be a color image. In embodiments with a monochromatic lightdetector 160, the LR projection image may be a black and white image.

Each sequence of sub-pixel shifted LR projection images can refer to nLR projection images sampled at n sampling times where neighboringprojection images in time are separated by less than a pixel size (i.e.sub-pixel shift). During a scanning cycle, n LR projection images (I₁, .. . , I_(n)) may be captured at n sequential sampling times (t₁, . . .t_(n)). Any suitable number, n (e.g., 1, 3, 5, 10, 100, etc.) of LRprojection images may be captured during a scanning cycle. Also, anysuitable number (e.g., 1, 3, 5, 10, 100, etc.) of sequences of sub-pixelshifted LR projection images may be captured by the light detector 160during a scanning cycle. If multiple sequences are captured, thesequences can include different groups of LR projection images or thesequences can overlap sharing one or more LR projection images. In oneexample, 9 LR projection images (I₁, I₂, I₃, I₄, I₅, I₆, I₇, I₈, I₉) maybe captured at 9 sequential sampling times (t₁, t₂, t₃, t₄, t₅, t₆, t₇,t₈, t₉). In an overlapping case of the above example, sequences couldbe: 1) I₁, I₂, I₆, and I₈, and, 2) I₆, I₇, I₈, and I₉. In anon-overlapping case, sequences could be: 1) I₁, I₂, I₃, and I₄, and 2)I₅, I₆, I₇, and I₈. In others examples, a sequence of sub-pixel shiftedLR projection images may be based on non-sequential sampling times. Forexample, 9 LR projection images (I₁, I₂, I₃, I₄, I₅, I₆, I₇, I₈, I₉) maybe captured at 9 sequential sampling times (t₁, t₂, t₃, t₄, t₅, t₆, t₇,t₈, t₉) and the sequence of projection images may be (I₆, I₂, I₉, I₁).

In embodiments, the light detector 160 may capture an LR projectionimage at each scanning time during a scanning cycle. For example, alight detector 160 may capture an LR projection image associated witheach scanning location in the scanning pattern shown in FIG. 4(b). Inthis example, the light detector 160 may capture a projection image ateach scanning time as the light element 117 moves through each rowsequentially of the two-dimensional array of scanning locations in thescanning pattern. If scanning locations in each row are associated withsub-pixel shifted projections 170, the light detector 160 may capture 15sequences of sub-pixel shifted projection images during the scanningcycle. In this case, each sequence is associated with a row of scanninglocations in the scanning pattern.

A motion vector can refer to the translational motion of projectionimages in a sequence of LR projection images, collectively termed themotion vector of the sequence of LR projection images. The motion vectoris based on the amount of shifting of the projection images at a plane.A motion vector of a sequence of sub-pixel shifted LR projection imagescan be calculated from the associated projection images captured by thelight detector 160. The motion vector may be calculated at any plane ofinterest. For example, the motion vector can be determined at the planeat the sensing surface 162. In this example, the motion vector isdetermined in terms of the local x′-axis and y′-axis at the sensingsurface 162 of the light detector 160. As another example, the motionvector can be calculated at other planes through an object 152 beingexamined. The planes through the object 152 may be parallel to the planeof the sensing surface 162 in some cases.

In embodiments, an HR image of a specimen 150 can be constructed using asuitable super resolution (SR) algorithm based on data associated with asequence of sub-pixel shifted LR projection images and a motion vectorof the sub-pixel shifted LR projections in the sequence. An example ofimage resolution obtainable by embodiments of the SPLM system 10 may beabout 0.66 micron.

An SR algorithm can refer to an image processing technique thatconstructs a HR image (e.g., sub-pixel resolution image) from a sequenceof sub-pixel shifted LR projection images. Any suitable SR algorithm canbe used by embodiments of the SPLM system 10. An example of a suitableSR algorithm is a shift-and-add pixel SR algorithm. Some examples ofsuitable SR algorithms can be found in Lange, D., Storment, C. W.,Conley, C. A., and Kovacs, G. T. A., “A microfluidic shadow imagingsystem for the study of the nematode Caenorhabditis elegans in space,”Sensors and Actuators B Chemical, Vol. 107, pp. 904-914 (2005)(“Lange”), Wei, L., Knoll, T., and Thielecke, H., “On-chip integratedlensless microscopy module for optical monitoring of adherent growingmammalian cells,” Engineering in Medicine and Biology Society (EMBC),2010 Annual International Conference of the IEEE, pp. 1012-1015 (2010)(“Wei”), Milanfar, P., “Super-Resolution Imaging”, CRC Press, (2010)(“Milanfar”), and Hardie, R., Barnard, K., and Armstrong, E., “Joint MAPregistration and high-resolution image estimation using a sequence ofundersampled images,” IEEE Transactions on Image Processing 6, pp.1621-1633 (1997) (“Hardie”), which are hereby incorporated by referencein their entirety for all purposes. An example of a suitable superalgorithm is the general pixel super resolution model and solutiondescribed in Section II.

The SPLM system 10 of FIG. 1 also includes a host computer 200communicatively coupled to the light detector 160. The host computer 200comprises a second processor 210 (e.g., microprocessor), a second CRM220, and an image display 230. The image display 230 and the second CRM220 are communicatively coupled to the second processor 210.Alternatively, the host computer 200 can be a separate device from theSPLM system 10. The host computer 200 can be any suitable computingdevice (e.g., smartphone, laptop, tablet, etc.)

The second processor 230 executes code stored on the second CRM 220 toperform some of the functions of SPLM 10 such as, for example:interpreting data from one or more sequences of sub-pixel shifted LRprojection images captured and communicated in one or more signals fromthe light detector 160, determining a motion vector of a sequence ofsub-pixel shifted projections, constructing a 2D HR image from dataassociated with a sequence of sub-pixel shifted LR projection images,constructing a 3D HR image from data associated with a sequence ofsub-pixel shifted LR projection images, displaying one or more HR imageson the image display 230, etc.

The second processor 210 can receive one or more signals with light dataand other data from the light detector 160. For example, the processor210 can receive one or more signals with light data associated with oneor more sequences of sub-pixel shifted LR projection images sampled at acorresponding sequence of n scanning times (t₁, t₂, t₃, . . . t_(n)).The second processor 210 can also determine a motion vector based on thesequence of sub-pixel shifted LR projection images. The second processor210 can also construct HR images and associated image data based thedetermined motion vector and data associated with at least one sequenceof sub-pixel shifted LR projection images. In some cases, theconstructed HR image of the object 150 is a black and white 2D/3D image.In other cases, the constructed HR image of the object 150 is a color2D/3D image.

In one embodiment, a HR color image can be generated by using differentwavelengths of illumination 118 at different sampling times to generatea multiple sequences of sub-pixel shifted LR projection images at alight detector 160. Each sequence is associated with a differentwavelength. The second processor 210 can generate HR color image andassociated image data based on the different sequences associated withdifferent wavelengths. For example, three wavelengths of light (e.g.,wavelengths associated with red, green, blue (RGB) colors) can besequentially generated by a light element 117 to generate threesequences of sub-pixel shifted projection images associated with threewavelengths of light. The processor 210 can combine the image data fromthe sequences associated with the different wavelengths to generatemulti-wavelength or color image data (e.g., RGB color image data). Themulti-wavelength or color HR image data can be used to generate amulti-wavelength or color HR image on the image display 230.

The second CRM (e.g., memory) 220 can store code for performing somefunctions of the SPLM system 10. The code is executable by the secondprocessor 210. For example, the second CRM 220 of embodiments mayinclude: a) code with a SR algorithm, b) code with a tomographyalgorithm, c) code for interpreting light data received in one or moresignals from the light detector 160, d) code for generating a 3D HRimage, e) code for constructing a color sub-pixel image, f) code fordisplaying SR two-dimensional and/or three-dimensional images, g) and/orany other suitable code for performing functions of the SPLM system 10.The second CRM 220 may also include code for performing any of thesignal processing or other software-related functions that may becreated by those of ordinary skill in the art. The code may be in anysuitable programming language including C, C++, Pascal, etc.

The SPLM system 10 also includes an image display 230 communicatively tothe processor 210 to receive data and provide output such as HR imagesto a user of the SPLM system 10. Any suitable display may be used. Forexample, the image display 230 may be a color display or a black andwhite display. In addition, the image display 230 may be atwo-dimensional display or a three-dimensional display. In oneembodiment, the image display 230 may be capable of displaying multipleviews of an object 150.

Modifications, additions, or omissions may be made to SPLM system 10 orthe SPLM device 100 without departing from the scope of the disclosure.In addition, the components of SPLM 10 or SPLM device 100 may beintegrated or separated according to particular needs. For example, thesecond processor 210 may be integrated into the light detector 160 sothat the light detector 160 performs one or more of the functions of thesecond processor 160 in some embodiments. As another example, the secondprocessor 160, second CRM 220, and image display 230 may be componentsof a computer separate from the SPLM system 10 and in communication withthe SPLM system 10. As another example, the second processor 160, secondCRM 220, and/or image display 230 may be integrated into parts of theSPLM device 100. For example, the image display 230 may be part of theillumination display 116, the first processor 112 and second processor210 may be integrated into a single processor, and/or the first CRM 114and second CRM 220 may be integrated into a single CRM.

II. Imaging Method Used by SPLM and e-Petri

A. Principle and Resolution

Nyquist criterion considerations dictate that the raw projection(shadow) image resolution from an image sensor (e.g., CMOS image sensor)may be no better than two times the pixel size. The SPLM and e-Petrisystems of embodiments of embodiments use a high sampling rate in thetime domain to offset the sub-Nyquist rate sampling in the spatialdomain of the projection images, combining work done in super resolutionimaging with advanced sensor (e.g., CMOS) technology to produce a lowcost, HR microscopy device with significant resolution enhancement.

The SPLM and e-Petri systems of embodiments include a thin transparentlayer 165 between the light detector 160 and the object 152 beingimaged. The transparent layer 165 separates the objects 152 (e.g.,cells) from the actual light sensitive region of the light detector 160(e.g., sensor chip). During scanning, the scanning illumination source110 shifts/scans or otherwise moves a light element 117 to differentpositions to provide illumination 118 (e.g., incoherent light) fromdifferent illumination angles above the specimen 150. The light detector160 acquires one or more sequences of LR projection images. With themovement of the illumination 118, the projection image shifts across thelight detecting elements 166 (e.g., sensor pixels), as shown in FIG. 3.The amount of shadow shift is proportional to the thickness of thetransparent layer 165 and the tilt/shift extent of the light element117. As long as the shift between each raw projection image in eachsequence of LR projection images is smaller than the physical size ofthe light detecting element (e.g., pixel size), the information frommultiple sub-pixel-shifted LR shadow images can be used to create asingle HR (sub-pixel resolution) image with a suitable super-resolutionalgorithm.

In previous super resolution microscanning systems, a specimen wasmounted to a stage and the stage was scanned in sub-pixel increments. Inthis prior approach, the position of the stage needed to be accuratelycontrolled in precise sub-pixel steps. Typically, controllers andactuators were used to control the required precise position of thestage. High precision meant high cost of setup and alignment wasrequired by these systems.

In a previous super resolution optofluidic system, optofluidics areincorporated to generate HR images from LR projection images in a highthroughput manner. In this system, an optofluidic sample-delivery schemeis employed to capture a sequence of images of the sample translatingacross a CMOS imaging sensor (pixel) array. The system usessuper-resolution processing techniques to achieve HR images from thesequences of LR projection images as described in U.S. patentapplication Ser. No. 13/069,651, which is hereby incorporated byreference in its entirety for all purposes, and described in Zheng. Thismethod relies upon capturing a sequence of LR projection images ofobjects (e.g., cells) as they flow through a fluid channel, across alight detector (e.g., CMOS imaging sensor array). However, imaging inthis system requires fluidic (e.g., microfluidic) flow of specimensacross a scanning area. Adherent, confluent, or contiguously arrangedspecimens are simply incompatible with imaging in a fluidic mode. Forexample, in order to make an object flow across the fluid channel, anobject cannot attach to the surface of image pixel (i.e. there isdistance between the object and the image pixel). Such a distanceresults in a blurry image of the object. In addition, the field of viewcan be limited by the geometry of the fluid channel.

The SPLM and e-Petri systems and devices use a scanning illuminationsource 110 to position a light element 117 over the specimen 150. Inthis approach, there may be no need for precise alignment. The scanningillumination source 110 is located at a larger distance from the sensingsurface 162 than the object 152. Thus, shifts of the light element 117correspond to smaller shifts of the projections 170 on the sensingsurface 162. The scanning illumination source 110 can control thesub-pixel shifts of the projections at the sensing surface 162 directlywith more controllable larger shifts of the light element 117 at thescanning illumination source 110. In this way, the scanning illuminationsource 110 can easily and accurately keep the projection shifts atsub-pixel values than previous systems such as microscanning systems,optofluidic systems, etc. Moreover, without the need of mechanicalscanning or microfluidic flow, the speed of scanning can be much faster.The scanning illumination source 110 can scan light at speeds up to therange of kHz. This is two orders of magnitude higher than priormechanical microscanning schemes. In addition, the cost of the devicescan be much lower since it uses a scanning illumination source 110 suchas a LED screen or LED matrix.

FIG. 5(a) is a projection image captured by a light detector 160 of anSPLM system 10 or an e-Petri system at a single sampling time, accordingto embodiments of the invention. In this example, the specimen 150 beingimaged includes a group of 3 μm microspheres. FIG. 5(b) is an HR imagereconstructed by the SPLM system 10 or the e-Petri system, according toembodiments of the invention. The system reconstructed the HR imagebased on data from a sequence of sub-pixel shifted LR projection imagesincluding the LR projection image shown in FIG. 5(a).

FIG. 6(a) is an LR projection image of a portion of a HeLa cell specimencaptured by a light detector 160 of an SPLM system 10 at a singlesampling time, according to embodiments of the invention. FIG. 6(b) isan HR image reconstructed by the SPLM system 10 or the e-Petri system,according to embodiments of the invention. The SPLM system 10 or thee-Petri system reconstructed the HR image based on data from a sequenceof 225 sub-pixel shifted LR projection images including the LRprojection image shown in FIG. 6(a).

FIG. 7(a) is a large field of view color HR image of a confluent HeLacell specimen 150 constructed by an SPLM system 10 or an e-Petri system,according to embodiments of the invention. The specimen 150 was stainedwith Giemsa. During reconstruction, each pixel at the LR projectionimage level (2.2 μm) was enhanced into a 13×13 pixel block in thereconstructed HR image. The color HR image contains about 8.45×10⁸pixels. The sensing area 164 (image area) was 6 mm×4 mm. A 15×15 arrayof scanning locations for each color illumination 118 was used. FIG. 7(b1) is an LR projection image from a small region of FIG. 7(a) and FIG.7(c 1) is an LR projection image from a small region of FIG. 7(b 1),captured by the light detector 160 of an SPLM system 10 or an e-Petrisystem, according to embodiments of the invention. FIG. 7(b 2) is areconstructed HR image from the same small region of FIG. 7(a) and FIG.7(c 2) is a reconstructed HR image from a small region of FIG. 7(b 2)constructed by an SPLM system 10 or an e-Petri system, according toembodiments of the invention. FIG. 7(d) is a conventional microscopyimage of similar cells using a microscope with 40×, NA=0.66 objectivelens. From the reconstructed HR images in FIGS. 7(b 2) and 7(c 2),organelles within the HeLa cell can be discerned such as multiplenuclear granules (indicated by red arrows), and the nucleus. Thereconstructed HR images also closely corresponded to conventionalmicroscopy images acquired from similar cells.

FIG. 8(a) is an HR image of a specimen 150 having 500 nm microspheres(Polysciences) as constructed by an SPLM system 10 or an e-Petri system,according to embodiments of the invention. The imaging process used toconstruct the HR image was identical the one used to construct the HRimages in FIG. 7. For a single 500 nm microsphere, the bright center ofthe microsphere was clearly resolved as shown in FIG. 8(a), with thefull-width at half maximum (FWHM) of 690 nm. FIG. 8(b) is an HR image ofa magnified small feature of the stained HeLa cell specimen 150 of FIG.7 as constructed by an SPLM system 10 or an e-Petri system, according toembodiments of the invention.

Since microscopy resolution may be defined in some cases based on agiven microscope's ability to resolve two closely spaced feature points,the case of two closely spaced microspheres can be analyzed to establisha resolution of an SPLM system 10 or an e-Petri system of embodiments.FIG. 8(a) shows the reconstructed images of two closely packed 500 nmmicrospheres with center-to-center distance of 660 nm. The data trace inFIG. 8(a) shows a valley between the two peaks and, thus, establishesthat the resolution may be 660 nm or better in some embodiments. Tofurther verify this point, FIG. 8(b) shows the magnified small featureof the stained HeLa cell specimen of FIG. 7 and the FWHM of this featurewas estimated to be about 710 nm.

B. Concept

In embodiments such as the example shown in FIG. 1, the specimen 150 isplaced on a specimen surface 140 located slightly above the activesensing area 164 of the sensing surface 162. The illuminating display116 (e.g., a monochromatic or color LCD) of a scanning illuminationdevice 110 (e.g., mobile communication device) is located at a distance,d, (e.g., about 5-10 mm) away from the sensing surface 162. A lightelement 117 (e.g., one or more light emitting components (e.g., pixels))of the illuminating display 117 provide illumination 118 (e.g.,incoherent light). The illumination 118 generates a projection 170(shadow) on the light detector 162. The light detector 160 can capturean LR projection image. This LR projection image is the best achievablegiven the size limitations (e.g., pixel size limitations) of the lightdetecting elements 166 (as shown in FIG. 2), but “low resolution” inthat features sizes of the specimen 150 may be much smaller than thesize (e.g., pixel size) of the light detecting element 166.

In embodiments, to improve the resolution, a sequence of sub-pixelshifted LR projection images is captured, for which light emittingcomponents (e.g., pixels) on the illuminating display 116 (e.g., an LCD)provide illumination 118. Each of these LR projection images is asub-pixel shifted projection image of the specimen 150. The sequence ofsub-pixel shifted LR projection images can be based on the scanninglocations of the light element 117 during a scanning cycle. For a knownsub-pixel displacement, these sub-pixel shifted LR projection images canbe used to create a HR (e.g., sub-pixel resolution) 2D image using pixelsuper-resolution techniques. This HR image can further be deconvolvedwith the point spread function of the pixel and optical system torecover a focused image of the specimen. SPLM system 10 e-Petri systemsmade possible precise scanning of the light element 117 in conjunctionwith pixel super-resolution image processing techniques.

Furthermore, this concept of imaging can be extended beyond twodimensions. Computed tomography using different incident angles of lightto generate multiple projections can be used to create a threedimensional reconstruction of the object. An example of using tomographyto generate a 3D image can be found in Miao, J. R. R. Qin, Tourovskaia,Anna, Meyer, Michael G., Neumann, Thomas, Nelson, Alan C., and Seibel,Eric J., “Dual-modal three-dimensional imaging of single cells withisometric high resolution using an optical projection tomographymicroscope,” J. Biomed., Opt., Vol. 14, 064034 (Dec. 21, 2009), which ishereby incorporated by reference in its entirety for all purposes. Inour scheme, the shifting light element 117 (e.g., sets of pixels) acrossthe illuminating display 116 (e.g. LCD) can provide different angles ofincident light necessary for 3D imaging.

C. Operating Principles

In one operation, the specimen 150 is placed slightly (e.g., in therange of several hundred nanometers to microns) above the sensingsurface 162 (e.g., outer surface) of the light detector 160 (e.g., CMOSimaging sensor array). Individuals or small sets of light emittingcomponents 166 (e.g., pixels) on the illuminating display (e.g., an LCD)are illuminated in succession to illuminate the specimen 150 at distance(e.g. 5 mm-10 mm) away, allowing the light detector 160 to record one ormore sequences of sub-pixel-shifted LR projection images, which are“pixilated.” One or more sequences of sub-pixel shifted LR projectionimages can be processed using super resolution techniques to combinemany LR projection images to create a smaller sequence of HR images. Anexample of a super resolution technique can be found in Richard, L. M.,Shultz, R., Stevenson, Robert L., “Subpixel motion estimation forsuperresolution image sequence enhancement,” Journal of VisualCommunication and Image Representation (1998), which is herebyincorporated by reference in its entirety for all purposes.

Super resolution or super resolution techniques refer to a general namefor the many promising new techniques for imaging processing that caninvolve creating a single HR image from a sequence of lower resolutionimages. Some super resolution techniques can be found in Park, SungCheol, Park, and Min Kyu, Kang, Moon Gi, “Super-resolution imagereconstruction: a technical overview,” IEEE Signal Processing Magazine,pp. 21-36 (May 2003) (“Park”), which is hereby incorporated by referencein its entirety for all purposes. The general principle involves takinga sequence of LR projection images in which the target is sampled atbelow the Nyquist rate, but for which subsequent frames involve a slightsub-pixel translational shift. This principle can be found in Russell,K. J. B., Hardie, C., Bognar, John G., Armstrong, and Ernest E., Watson,Edward A., “High resolution image reconstruction from a sequence ofrotated and translated frames and its application to an infrared imagingsystem,” Optical Engineering (1997), which is hereby incorporated byreference in its entirety for all purposes. If this translational shiftis known, then a system of matrix equations can be established from thelower resolution sequence to solve for sub-pixel values to create asingle HR image. In general, the original HR image can theoretically berecovered even from a significantly decimated, blurred, translated, androtated lower resolution image sequence; resolution is limited only bythe diffraction limit and noise, as described in Park.

D. Flowchart of Exemplary Method of Operation

FIG. 9 is a flow chart of an exemplary operation of an SPLM device 100or an e-Petri device, according to embodiments of the invention. TheSPLM device 100 and e-Petri device include a scanning illuminationsource 110 for shifting or otherwise positioning a light element 117 toprovide illumination from different illumination angles to an object 152being imaged. Although imaging of an object 152 is described in thissection, the method can be used to image a specimen 150 having anysuitable number of objects 152. The SPLM device 100 and the e-Petridevice also include a specimen surface 140, a light detector 160 havinga sensing surface 162, a thin transparent layer 165 between the sensingsurface 162 and the specimen surface 140, and a processor (firstprocessor 112 and/or second processor 210). The scanning illuminationsource 110 is located at a distance, d, from the sensing surface 162.This exemplary operation includes an imaging run having a singlescanning (illumination) cycle. Other embodiments may have an imaging runwith multiple scanning (illumination) cycles.

In step 310, the object 152 is placed onto the specimen surface 140 ofthe SPLM 100 or the e-Petri device. The object 152 may be locatedproximal a sensing area 164 of the sensing surface 162 at the activelayer of the light detector 160.

In step 320, the processor determines a scanning pattern. An example ofa scanning pattern is shown in FIGS. 4(a) and 4(b). The scanning patternmay include scanning locations at different times during a scanningcycle and properties (e.g., wavelength(s) of light used, the size andshape of the light element, the intensity(ies) of the light element,etc.) of the light element 117 at different scanning locations, theamount of sub-pixel shift desired between subsequent LR projectionimages, the total number of LR projection images desired in the scanand/or in the sequence of LR projection images, the total number of HRimages desired in an imaging run, and other suitable information relatedto the operation of the SPLM system 10 or e-Petri system. The processormay retrieve a predetermined scanning pattern from the CRM (first CRM114 or second CRM 220) or the processor may determine a scanning patternbased on input from a user of the SPLM system 10 or the e-Petri system.For example, the user may provide information such as properties of thelight element, the amount of sub-pixel shift desired between subsequentLR projection images, the total number of HR images desired, and othersuitable input.

The scanning locations in the scanning pattern can be determined togenerate a sequence of sub-pixel shifted projections at the sensingsurface 162. The shift of a projection 170 is proportional to thethickness of the transparent layer 165 and the tilt/shift extent (i.e.distance or illumination angle change between neighboring scanninglocations). The amount of translation of the light element 170 betweenneighboring scanning positions that will result in sub-pixel shifting ofthe projections 170 can be determined based on the thickness of thetransparent layer 165 and the required sub-pixel shift value. Thescanning locations in the scanning pattern can be based on the amount oftranslation of the light element 170 between neighboring scanningpositions.

In step 330, the scanning illumination source 110 scans or otherwisepositions the light element 117 and modifies the properties of the lightelement 117 according to the scanning pattern. In one embodiment, thescanning illumination source 110 (e.g., smartphone) has an illuminatingdisplay 116 in the form of an LCD. In this example, the scanning patternmay include a two-dimensional array of scanning positions and thescanning times associated with the scanning positions. During scanningthe light element 117 may be a set of light emitting components (e.g.pixels) in the LCD which are sequentially illuminated to shift the lightelement 117 through each row/column of the two-dimensional array ofscanning locations. The properties of the light element 117 may vary atdifferent locations. For example, the size (number of light emittingcomponents) of the light element 117 may vary to maintain approximatelythe same intensity level at a location at the sensing surface 162.

In one embodiment, the light element 117 can provide illumination 118 ofn different wavelengths λ₁, . . . , λ_(n) at different times during ascanning cycle to obtain a sequence of projection images for eachwavelength. Any suitable number of wavelengths may be used (e.g., n=1,2, 3, 4, 5, . . . , 20). In one embodiment, the light element 117 mayprovide illumination 118 of three wavelengths λ₁, λ₂, and λ₃corresponding to red, green, blue colors at different sampling times. Insome cases, the illumination 118 from one scanning location to aneighboring scanning location may have different wavelengths. In othercases, the illumination 118 may have a first wavelength during a firstseries of scanning positions, and then provide illumination 118 of asecond wavelength during a second series of scanning positions, and soforth until n sequences of projection images corresponding to ndifferent wavelengths have been captured.

In step 340, the light detector 160 captures one or more sequences ofsub-pixel shifted LR projection images of the object 152 as the lightelement 117 translates to different scanning positions. The lightdetector 160 captures an LR projection image for each scanning location.Any suitable number of images (e.g., 3, 5, 10, 50, 100, etc.) may be ineach sequence. In one example, the scanning locations may be in the formof a two-dimensional array of scanning positions, where each row/columnof scanning positions can generate a row of sub-pixel shiftedprojections 170. In this example, a sequence of sub-pixel LR projectionimages may be captured as the light element 117 shifts across eachrow/column in the two-dimensional array of scanning positions.

In step 350, the processor uses a suitable method to determine themotion vector of the projections 170 at the sensing surface 162. In somecases, the processor may also determine motion vectors of theprojections 170 at other parallel planes through the object 152. Anysuitable method of determining a motion vector can be used. In oneexample, the motion vector of the projections at the sensing surface 162may be determined based on the distance between neighboring scanningpositions and the thickness of the transparent layer 165. In anotherexample, the motion vector of the projections at planes parallel to thesensing surface 162 may be determined based on the distance betweenneighboring scanning positions and the thickness of the transparentlayer 165 and the distance between the plane and the sensing surface162.

In step 360, the processor uses an appropriate SR algorithm to constructa HR image of the object 152 from data from a sequence of sub-pixelshifted LR projection images and corresponding motion vector(s). Forexample, the processor can construct a 2D image of the object 152 at aplane through the object 152 by using a motion vector at that plane. Inone example, the processor can generate a 3D HR image by stacking the 2DHR images constructed based on motion vectors at different planes. Ifthe light detector 160 is a monochromatic light detector, the HR imagewill be a monochromatic HR image (black and white HR image). If thelight detector 160 is a color light detector (e.g., a color CMOS imagingsensor), the image resulting from this reconstruction is a color image.

In one embodiment, a shift-and-add SR algorithm may be used to constructan HR image with data from a sequence of sub-pixel shifted LR projectionimages. In this embodiment, an HR image grid is formed with anenhancement factor of n, where each n-by-n pixel area of the HR imagegrid corresponds to a 1-by-1 pixel area of the LR frame grid. Then, theHR image grid is filled with the corresponding pixel values from thesequence of sub-pixel shifted LR projection images. The mapping of thepixels within the n-by-n grid is determined from the known, estimatedsub-pixel shift of each image from the motion vector determined. Inother words, each LR projection image is shifted by the relativesub-pixel shift of the object 152 from its original position and thenadded together to form a HR image. Finally, deblurring using the wienerdeconvolution method may be used to remove blurring and noise in thefinal HR image.

In one embodiment, the light element 117 can provide illumination 118 ofn different wavelengths λ₁, . . . , λ_(n) at different times during ascanning cycle to obtain a sequence of projection images for eachwavelength. In this embodiment, the processor can use an appropriate SRalgorithm to reconstruct an HR image for each wavelength or color basedon each sequence of sub-pixel shifted LR projection images and themotion vector. The SPLM device 100 or e-Petri device can combine the HRimages of different wavelengths or colors to obtain a computed color HRimage. For example, an SPLM device 100 or e-Petri device using RGBillumination from can be used to construct a computed color (RGB) HRimage.

In step 370, the processor can display one or more HR images to asuitable image display 230 (e.g., two-dimensional display (color ormonochromatic), three-dimensional display (color or monochromatic)). Anysuitable image generated by the SPLM device 10 or e-Petri device can bedisplayed. Some examples of suitable images include: LR projectionimages, 2D black and white HR images, 2D color HR images, 3D black andwhite HR images, and/or 3D color HR images.

E. A Super Resolution Model and Solution

Embodiments of the SPLM system 10 and the e-Petri system use a SRalgorithm to reconstruct an HR image. One example of a SR algorithm isthe general pixel super resolution model and solution described in thisSection. This general pixel super resolution model and solution has asimple, fast and non-iterative method that preserves the estimationoptimality in the Maximum-Likelihood sense. Some details of thissuper-resolution model and solution can be found in Hardie, Elad, M.,and Hel-Or, Y., “A fast super-resolution reconstruction algorithm forpure translational motion and common space-invariant blur,” IEEETransactions on Image Processing, Vol. 10, pp. 1187-1193 (2001)(“Elad”), Farsiu, Sina, et al., Fast and robust multiframe superresolution,” IEEE Trans Image Process, vol. 13, pp. 1327-1344 (2004),and Farsiu S, et al., “Multiframe demosaicing and super-resolution ofcolor images,” IEEE Trans Image Process, vol. 15, pp. 141-159 (2006),which are hereby incorporated by reference in their entirety for allpurposes.

In an scanning cycle, a sequence of N captured LR projection images,Y_(k) (k=1, 2 . . . N) can be used to reconstruct an improved HR image,X. The images may be represented by lexicographically ordered columnvectors. The LR projection image can be modeled by the followingequation:Y _(k) =DHF _(k) X+V _(k)(k=1,2 . . . N)  (Eqn. 1)The matrix F_(k) stands for the sub-pixel shift operation for the imageX. The matrix H is the pixel transfer function of the light detector 160(e.g., CMOS image sensor). The matrix D stands for the decimationoperation, representing the reduction of the number of observed pixelsin the measured images. V_(k) represents Gaussian additive measurementnoise with zeros mean and auto-correlation matrix: W_(k)=E{V_(k)V_(k)^(T)}.

The Maximum-Likelihood estimation of X can be described as the followingexpression:

$\begin{matrix}{\overset{\Cap}{X} = {{Arg}\;{Min}\left\{ {\sum\limits_{k = 1}^{N}\;{\left( {Y_{k} - {{DHF}_{k}X}} \right)^{T}{W_{k}^{- 1}\left( {Y_{k} - {{DHF}_{k}X}} \right)}}} \right\}}} & \left( {{Eqn}.\mspace{14mu} 2} \right)\end{matrix}$And the closed-from solution for {circumflex over (X)} is shown to be:{circumflex over (X)}=H ⁻¹ R ⁻¹ P  (Eqn. 3)

where, R=Σ_(k=1) ^(N)F_(k) ^(T)D^(T)DF_(k), P=Σ_(k=1) ^(N)F_(k)^(T)D^(T)Y_(k)

R can be a diagonal matrix and the computation complexity of thisapproach may be: O(n*log(n)).

F. Different Schemes for the SPLM and e-Petri Systems and Devices

Scheme 1-2D Monochromatic Imaging

In a first scheme, SPLM systems 10 or e-Petri systems of embodiments maybe designed to generate 2D monochromatic HR images of a specimen 150using a suitable SR algorithm based on a sequence of LR projectionimages and a motion vector. For this case, there is only the knowntranslational shift and space invariant point spread function of thesystem, H, which is also known. Hence, more effective andcomputationally efficient super resolution techniques can be applied,such as the following as proposed in Elad. For an original HR image, X,of a specimen 150 that is the desired output of the SPLM system 10 orthe e-Petri system, a lower resolution image sequence of the sample:Y _(k) =DHF _(k) X+V _(k)(k=1,2 . . . N)  (Eqn. 4)is obtained, where F_(k) is the translational shift, H is the pointspread function of the optical system, D_(k) is the downsampling of theoriginal LR projection image and V_(k) is white noise withauto-correlation: W_(k)=E{V_(k)V_(k) ^(T)}. Hence, by minimizing theleast square error, the computed HR image {circumflex over (X)} isobtained from a sequence of N LR projection images as follows:

$\begin{matrix}{\overset{\Cap}{X} = {{Arg}_{X}\;{Min}\left\{ {\sum\limits_{k = 1}^{N}\;{\left( {Y_{k} - {D_{k}\; H\; F_{k}X}} \right)^{T}{W_{k}^{- 1}\left( {Y_{k} - {D_{k}H\; F_{k}X}} \right)}}} \right\}}} & \left( {{Eqn}.\mspace{14mu} 5} \right)\end{matrix}$

This optimization can be done computationally with iterative methodsdescribed in Elad. The end result of this optimization can be anin-focus HR image or sequence of HR images of the specimen generatedfrom the original LR projection images captured by the light detector160. (e.g., CMOS image sensor).

In embodiments, an SPLM system 10 or an e-Petri system may include anilluminating display 116 in the form of a pixel array (e.g., rectangularpixel array). For example, the illuminating display 116 may be arectangular pixel array of an LCD. In these embodiments, the sub-pixelshifts between subsequent LR projection images of a sequence may berelated by a characteristic sub-pixel spacing, α, related to theillumination scanning sequence, the detector array pixel sizes, and thedistances between the specimen 150 and source/detector. The distancebetween the specimen 150 and the illumination source 110 may be thedistance, d, between the top of the transparent layer 140 and thedisplay surface 119. The distance between the specimen 150 and the lightdetector 160 may be the thickness of the transparent layer 165.

For an SPLM system 10 or e-Petri system having an illumination display116 with a display surface 119 parallel to the sensing surface 162 ofthe light detector 160 and the specimen surface 140, the projection of apoint on the plane of the specimen surface 140 onto the detection plane(i.e. plane of the sensing surface 162) will be shifted in incrementsrelated to sin θ. The angle, θ is the angle of a line from the lightelement 117 (e.g., the center of a set of illuminated pixels on an LCD)to the point of the specimen 150, with respect to the specimen surfaceplane normal vector. For small angles, the sub-pixel shifts can beapproximated as equal and the solution for the motion vector of the LRsequence can be found by a simple one-dimensional optimization of, α. Incases where the illumination (LCD) plane and detector planes areparallel, the sub-pixel shifts should be ‘exactly’ equal.

Scheme 2—2D Color Imaging

In a second scheme, an SPLM system 10 or e-Petri system of embodimentsmay be designed to generate 2D color HR images of a specimen 150 using asuitable SR algorithm based on a sequence of LR color projection imagescaptured by a color light detector 160. In this scheme, the SPLM system10 or the e-Petri system includes a color light detector 112 (e.g., acolor CMOS sensor) that can capture a sequence of sub-pixel shiftedcolor LR projection images. The processor 210 can generate one or moreSR color images using a suitable color super resolution technique withthe sequence of sub-pixel shifted color LR projection images. Thesimplest technique involves using a monochromatic super resolutiontechnique on each of the color components independently. In anotherexample, a more complicated super resolution technique can be used thatinvolves transforming to a different color space, such as the one foundin Farsiu, Sina, et al., “Advances and challenges in super-resolution,”Wiley Periodicals (2004), which is hereby incorporated by reference inits entirely for all purposes.

Scheme 3—2D Computed Color Imaging

In a third scheme, an SPLM system 10 or e-Petri system of embodimentsmay be designed to generate 2D color HR images of a specimen 150 using asuitable SR algorithm based on multiple sequences of LR frames, eachsequence associated with a different wavelength or color of illumination118. The SPLM system 10 or e-Petri system can construct a 2D color HRimage based on each sequence associated with a differentwavelength/color. The SPLM system 10 can combine the 2D color HR imagesassociated with the different wavelengths to create a 2D multi-color HRimage of the specimen 150. The SPLM system 10 or e-Petri system of theseembodiments includes a scanning illumination source 100 with a colorillumination display 116 (e.g., color LCD) or another device that cangenerate color illumination 118. Any suitable wavelengths and number ofwavelengths may be used. In one example, wavelengths of light may bechosen that cover the widest viewable color range. In some cases,separate scans using different wavelengths/colors can be used to captureseparate RGB sequences of projection images. In other cases, the lightelement 117 may sequentially alternate between the differentwavelengths/colors in a single scan.

In one embodiment, the SPLM system 10 or e-Petri system may include ascanning illumination source 100 having a RGB illumination display 116(e.g., a RGB LCD). In this embodiment, separate red, green, and blue(RBB) scans can be used to capture separate RGB sequences of LRprojection images (i.e. red sequence, green sequence, and bluesequence). The SPLM system 10 or e-Petri system of this embodiment cangenerate an HR RGB image based each sequence. The SPLM system 10 ore-Petri system can combine the 2D color HR images based on each sequenceto generate a RGB image.

Scheme 4—3D Imaging with 3D Display

In a fourth scheme, an SPLM system 10 or e-Petri system of embodimentsmay be designed for 3D imaging on a 3D display 230. In this scheme, theSPLM system 10 or e-Petri system can generate n 2D HR images at ndifferent incidence angles to generate different views of the object 152based on the different locations of the light element 117.

In this scheme, the scanning illumination source 110 scans the lightelement 117 to locations that generate illumination 118 fromillumination angles in a range around each of the n different incidenceangles of interest. For example, if a view of the object 152 from 30 isdesired, the scanning illumination source 110 may scan the light element117 to generate illumination 118 from illumination angles in the rangeof 30+/−2 degrees in X/Y. As another example, if a view of the object152 from −30 degrees is desired, the scanning illumination source 110may scan the light element 117 to generate illumination 118 fromillumination angles in the range of −30+/−2 degrees in X/Y. The“angular” scan range for a single HR image may be constant and small (4degrees in this example) relative to the large angle displacements usedto get different views for 3D imaging. Each of the HR images is stillobtained from reconstructing from an LR projection image sequence,captured by scanning the illumination, but at a much larger angle away.

The 2D HR images from different incidence angles can be combined anddisplayed on a 3D display 230 (e.g., 3D monitor), or as a rotating gifor video file. This can be achieved by using different regions of theillumination LCD to generate high resolution projection images of asample, but from different angles.

In imaging schemes where a view at a plane parallel to the sensingsurface may be desired, the scanning illumination source 110 may scanthe light element 117 to locations that generate illumination 118 fromillumination angles in a range around normal to the sensing surface. Forexample, the scanning illumination source 110 may scan the light element117 to generate illumination 118 from illumination angles in the rangeof +/−2 degrees in X/Y.

Scheme 5—3D Focusing

In a fifth scheme, an SPLM system 10 or e-Petri system of embodimentsmay be designed to “focus” 2D HR images at different planes of interestthrough the specimen 150. The SPLM system 10 or e-Petri system can alsostack the “focused” 2D HR images at different planes to generate a 3D HRimage. For a three-dimensional specimen, the SPLM system 10 or e-Petrisystem can construct HR images from sequences of sub-pixel shifted LRprojection images based on different motion vectors associated withdifferent sub-pixel shifts in order to achieve “focusing” at differentplanes within the three-dimensional sample.

Under this scheme, the SPLM system 10 or e-Petri system can constructeach focused 2D image at a plane based on a captured sequence ofsub-pixel shifted LR projection images and the determined motion vectorat the plane. For example, the SPLM system 10 or e-Petri system maycreate a 2D HD image of a slice of a specimen 150 at a plane. In thisexample, the SPLM system 10 or e-Petri system determines the motionvector of the LR projection images at that plane. The SPLM system 10 ore-Petri system constructs the focused 2D HD image based on thedetermined motion vector at the plane of interest and a sequence ofsub-pixel shifted LR projection images captured by the light detector160. The SPLM system 10 or e-Petri system can also refocus at multipleplanes by constructing HR images using multiple motion vectors and thesame sequence of sub-pixel shifted LR projection images.

Since the quality of the focus of the reconstructed image depends on thecorrect estimation of the sub-pixel shifts of the LR projection images,and these sub-pixel shifts depend on the distance of the specimen 150between the light detector 160 and the illumination planes, usingdifferent sub-pixel shifts (i.e. motion vectors) in the reconstructionstep can allow for refocusing to specific specimen planes above thelight detector 160. This effectively allows for a single, extensive scansequence of LR projection images to not only provide three dimensionaldata with projection images from different angles (previous scheme), butalso focusing to specific three dimensional planes.

In one embodiment, the scanning illumination device 110 sweep the lightelement 117 to generate illumination 118 between a wide range ofillumination angles in order to generate an extensive scan sequence ofLR projection images. FIG. 10 is a schematic drawing of threeprojections on a light detector 160 from three wide ranging incidenceangles, θ₁, θ₂, and θ₃, according to an embodiment of the invention.Changing the illumination angle of the light 118 from the light element117 can generates a sequence of three projections associated withdifferent views View 1, View 2, and View 3 of the object 152. In FIG.10, θ₁=0 degrees, and is in the direction of a negative z-axis. Thelight detector 160 can capture a sequence of LR projection imagesassociated with the shifting projections. The light detector 160 canalso capture multiple sequences of sub-pixels LR projection imagesassociated with the illumination sweeping between the wide rangingincidence angles. This extensive scan sequence of LR projection imagesmay be used to generate 3D data with projection images from thedifferent views (previous scheme), but also to provide focusing tospecific 3D planes.

III. E-Petri

Conceptually, the method of microscopy imaging used by the e-Petrisystem is simple to understand. Geometrically, a specimen (e.g., cellsbeing cultured) is placed directly on the surface of a light detector(e.g. a CMOS image sensor) or on the surface of a transparent layerlying over the light detector. If an idealized image sensor with a highdensity grid of infinitesimally small pixels were used and the specimenwere located directly on the image sensor surface, the idealized imagesensor would be able to collect a shadow (projection) image of thespecimen with excellent acuity. Unfortunately, currently availablesensor chips have relatively large pixels (e.g., 2.2 microns). Thisimplies that the direct shadow images of microscopic objects collectedby conventional sensor chips are intrinsically coarse. Specifically, theraw shadow image resolution would be no better than two times the pixelsize (as dictated by Nyquist criterion considerations). To address this,the following approach is taken to improve resolution or, morespecifically, to generate a denser grid of smaller ‘virtual’ pixels.

First, it is noted that there may be a transparent layer (e.g., thinpassivation layer) that separates the specimen from the actual lightsensitive region of the sensor chip. With this recognition in mind,incoherent illumination is sequentially tilted/shifted above thespecimen and a sequence of raw images may be acquired. With theincremental tilt/shift of the illumination, the target specimen's shadowwill incrementally shift across the sensor pixels as shown in FIG. 3.The amount of shadow shift is proportional to the transparent layerthickness and the tilt/shift of the illumination source. As long as theshadow shift between each raw image frame is smaller than the physicalpixel size, the information from sequence of sub-pixel-shifted shadowimages can be combined to create a single HR image with a suitable pixelsuper-resolution algorithm. Some examples of super resolution imagingand super resolution algorithms can be found in Milanfar, P.,“Super-Resolution Imaging,” CRC Press, (2010), Hardie, R., Barnard, K. &Armstrong, E., “Joint MAP registration and high-resolution imageestimation using a sequence of undersampled images,” IEEE Transactionson Image Processing 6, pp. 1621-1633 (1997), Elad, M. & Hel-Or, Y., “Afast super-resolution reconstruction algorithm for pure translationalmotion and common space-invariant blur,” IEEE Transactions on ImageProcessing 10, pp. 1187-1193 (2001), Farsiu, S., Robinson, M., Elad, M.& Milanfar, P., “Fast and robust multiframe super resolution,” IEEETransactions on Image Processing,” 13, pp. 1327-1344 (2004) (“Elad”),Farsiu, S., Robinson, D., Elad, M. & Milanfar, P., “Advances andchallenges in super resolution,” International Journal of ImagingSystems and Technology 14, 47-57 (2004), and Farsiu, S., Elad, M. &Milanfar, P., “Multiframe demosaicing and super-resolution of colorimages,” IEEE Transactions on Image Processing 15, pp. 141-159 (2006),which are hereby incorporated by reference in their entirety for allpurposes. An example of super resolution model and solution that uses asuitable super resolution algorithm is described in Section IIE.

A. E-Petri System

FIG. 11 is a schematic diagram of an e-Petri system 600 having n e-Petridevices 610 and an expanded view of one of the e-Petri devices 610,according to embodiments of the invention. n can be any suitable numbersuch as 1, 2, 3, 4, 10, 20, etc. Each e-Petri device 610 of the e-Petrisystem 600 includes a scanning illumination source 110, an e-Petri dish620, and support 180 for holding the scanning illumination source 110 ina fixed position relative to the light detector 160. The e-Petri system600 also has a relay 700 (e.g., multiplexer), an incubator 800 formaintaining a predefined environment, and a host computer 200. The relay700 is in electronic communication with the n e-Petri devices 610 toreceive data. The host computer 200 is in electronic communication withthe relay 700 to receive the data relayed and controlled (e.g.,multiplexed) by the relay 700 from the n e-Petri devices 610. The ne-Petri devices 610 are located within a chamber formed by the walls ofthe incubator 800. The host computer 200 includes a second processor 210(shown in FIG. 1), a second CRM 220 (shown in FIG. 1), and an imagedisplay 230. The image display 230 and the second CRM 220 in electroniccommunication with the second processor 210. In some cases, there maynot be a second processor 210 or second CRM 220 and the functions ofthose components may be performed by one or more of the first processors112 or first CRMs 114.

In FIG. 11, the e-Petri devices 610 include a scanning illuminationsource 110 capable of providing illumination 118 to a specimen 150 froma plurality of illumination angles. The scanning illumination source 110includes a first processor 112, a first computer readable medium (CRM)114, and an illuminating display 116 (e.g., an LCD, a light emittingdiode (LED) display, etc.). In some cases, the first processor 112 andfirst CRM 114 may be separated from the scanning illumination source110. The first processor 112 is in electronic communication with theilluminating display 116 and with the first CRM 114. The illuminatingdisplay 116 includes a light element 117 (e.g., a set of one or moreilluminated pixels in an LCD/LED) providing illumination 118 (e.g.,incoherent light). The illuminating display 116 also includes a displaysurface 119. The display surface 119 is located at a distance, d, fromthe sensing surface 162. The light element 117 is located at the displaysurface 119 in the illustrated example. In other embodiments, atransparent layer may be located between the display surface 119 and thelight element 117 or may be located outside the display surface 119. Thescanning illumination source 110 also includes an x-axis, a y-axis (notshown), and a z-axis. The x-axis and y-axis lie in a plane at thedisplay surface 119. The z-axis is orthogonal to this plane. As shown bya dotted line, the light detector 160 may optionally be in electroniccommunication with the first processor 112 of the scanning illuminationsource 110 to synchronize operations.

In FIG. 11, the e-Petri device 610 also includes an e-Petri dish 620.The e-Petri dish 620 includes a light detector 160 having a sensingsurface 162 and a transparent layer 165 lying over the light detector160 (e.g., commercially available CMOS image sensor having 2.2 micronpixels). In other cases, the transparent layer 165 may be part of thelight detector 160. The sensing surface 162 includes a sensing area 164(e.g., 6 mm×4 mm area). The transparent layer 165 includes a specimensurface 140 for receiving a specimen 150 (e.g., cell culture) and othermaterials (e.g., culture medium). The light detector 160 also has anx′-axis, a y′-axis (not shown), a z′-axis. The x′-axis and y′-axis liein a plane at the sensing surface 162 of the light detector 160. Thez′-axis is orthogonal to this plane. The e-Petri dish 620 also includesan optional well 170 having a peripheral wall 172 and an optional cover176.

In FIG. 11, a specimen 150 with five objects 152 (e.g., cells) islocated on the specimen surface 140 within the wall 172 of the well 170.Although five objects 152 are shown, the specimen 150 of otherembodiments may have any suitable number (e.g., 1, 2, 10, 100, 1000,etc.) of objects 152 or portions (e.g., cell components) of objects 152.

The e-Petri system 600 in FIG. 11 may function as a multimodal on-chipimaging system with a multiplicity of functions, uses, and benefits.This system can be made in a low-cost and compact manner and canincorporate the ability to grow cells or other objects 152 on componentsof the system itself. For example, the e-Petri dish 620 can include asimple chamber design with a medium affixed to the light detector 160where objects 152 (e.g. cells) can be cultured and stored. Multiplee-Petri devices 610 can be placed in a single incubator 800, to allowvarious functions to be performed, and different types of data to begenerated, simultaneously. For example, different e-Petri devices 610can use light elements 117 with different characteristics (e.g.wavelength, intensity), or filters to allow a user to image objects 152in both bright-field and fluorescence simultaneously. Multiple arrays ofchambers or a fluidic network can also be designed to provide control ofchemical and mechanical environment (not shown). Thus, this imagingsystem may be able to replace imaging systems with conventional Petridishes and well-plates in biology labs.

FIG. 12 is a photographic image of an e-Petri system 600 having a singlee-Petri device 610, according to embodiments of the invention. Thee-Petri system 600 includes the e-Petri device 610, an incubator 800,and a host computer 200 in communication with the e-Petri device 610.The e-Petri device 610 includes a illumination source 110 in the form ofa smartphone, an e-Petri dish 620 and a support 180 holding theillumination source 110 and the e-Petri dish 620. The e-Petri device 610is located within the incubator 800 to control the environment. The hostcomputer 200 includes a second processor 210 (shown in FIG. 1), a secondCRM 220 (shown in FIG. 1), and an image display 230. The host computer200 is in electronic communication with the e-Petri device 610 toreceive image data to display an image 231 of a specimen 150 (not shown)on the display 230.

During an exemplary operation of an e-Petri system 600 of FIGS. 11 and12, the illumination source 110 provides illumination 118 from differentillumination angles at different illumination times to generatesub-pixel shifted light projections 170 (as shown in FIG. 3) of thespecimen 150 at the sensing surface 162 of the light detector 160. Thelight detector 160 samples (captures) one or more sequences of sub-pixelshifted projection images of the specimen 150 at the differentillumination times. The second processor 210 of the host computer 200may receive data associated with the one or more sequences of sub-pixelshifted projection images. The data is relayed through the relay 700from the light detector 150. The second processor 210 may determinemotion vector of the projections 170 from one or more sequences ofsub-pixel shifted projection images. The second processor 210 can use asuitable super resolution algorithm to generate one or more HR images ofthe specimen 150 based on the motion vector and data of one or moresequences of sub-pixel shifted projection images. The one or more HRimages 230 and other related images 232 can be displayed on the display230.

An imaging run of an e-Petri system 600 can refer to a time intervalduring which operations of the e-Petri system 600 generate a sub-pixelresolution image of a specimen 150 or a portion of a specimen 150located at one or more of the e-Petri devices 610. An illumination cycleof an e-Petri device 610 can refer to the time interval during which thescanning illumination source 110 provides illumination 118 from aplurality of illumination angles corresponding to a plurality ofillumination times. Any suitable number (e.g., 1, 10, 100, 1000, etc.)of illumination times and corresponding illumination angles can be used.The plurality of illumination angles may be designed to generate asequence of sub-pixel shifted projections 170 (as shown in FIG. 3) ofthe specimen 150 on the sensing surface 162 of the light detector 160.The light detector 160 may sample a light projection 170 at samplingtimes corresponding to the illumination times. In FIG. 11, the lightelement 117 is shown at a single illumination time during anillumination cycle corresponding to a sampling time. In embodiments, ane-Petri system 600 can be designed to be automated to image (automatedlyimage) the specimen 150 periodically or otherwise repeatedly over a longterm. In these cases, the e-Petri system 600 performs multiple imagingruns over a longer term. For example, the e-Petri system 600 may bedesigned to perform periodic imaging of a cell culture on an hourlybasis over two-weeks.

FIG. 13(a) is a photographic image of an e-Petri dish 620 according toan embodiment of the invention, and a quarter for size comparison. Thee-Petri dish 620 includes a light detector 160, a transparent layer 165,and a well 170. The light detector 160 is in the form of a commerciallyavailable CMOS image sensor chip with a 6 mm×4 mm imaging area filledwith 2.2 micron pixels. The microlens layer and color filter on theimage sensor chip were removed to provide direct access to the imagesensor pixels (light detecting elements 166). The microlens layer andcolor filter were removed by treating the sensor chip under oxygenplasma for 10 min (80 W). The transparent layer 165 in the form of athin PDMS layer was prepared by mixing 1:10 with base and curing agent,then spin coated onto the sensing surface 162 followed by baking at 80°C. for 1 hour. The well 170 is a plastic square well comprising aperipheral wall 172 glued at the inner edges to the transparent layer165 of the light detector 160 with poly-dimethylsiloxane (PDMS). Thee-Petri dish 620 also includes a cover 176 hinged to an outer edge ofthe peripheral wall 172. In FIG. 13(a) a pipette is shown introducing aspecimen 150 into the well 170.

FIG. 13(b) is a photographic image of a partially disassembled e-Petridevice 610 having the e-Petri dish 620 of FIG. 13(a), according to anembodiment of the invention. As shown, the e-Petri device 610 includes ascanning illumination source 110 in the form of a smartphone with anilluminating display 116 in the form of a LED screen. The e-Petri device610 also includes the e-Petri dish 620 of FIG. 13(a). The e-Petri device610 also includes a support 180 for holding the scanning illuminationsource 110 in a fixed position at a distance 2.0 cm away from the lightdetector 160. In the illustrated example, the support 180 is made ofbuilding blocks that house the image sensor socket board and thesmartphone. The parallel alignment between the display surface 119 andthe sensing surface 162 may not be a critical consideration. In FIG.13(b), e-Petri device 610 is shown partially disassembled with theillumination source 110 separate from the other components of thee-Petri device 610.

FIG. 13(c) is a photographic of the assembled e-Petri device 610 of FIG.13(b), according to an embodiment of the invention. In FIG. 13(c), theillumination source 110 is located in the support 180. In an imaging runusing the e-Petri device 610 of the illustrate example, the illuminationsource 110 provided illumination 118 from perspective illuminationangles ranging from −60 degree to +60 degrees with respect to thesensing surface 162 of the light detector 160. The entire e-Petri device610 can be placed in an incubator 800 for automatic long term cellimaging and tracking.

The e-Petri device 610 of embodiments is an on-chip imaging device wherea specimen 150 such as cells and a culture medium can be stored andimaged. This device may be suitable to replace conventional microscopydevices having petridishes and well-plates in biology labs. The e-Petridevice 610 of embodiments may include any suitable combination ofstructures and devices for storing and imaging the specimen 150. InFIGS. 11, 13(a), and 13(b), the e-Petri device 610 includes anillumination source 110, an e-Petri dish 620, and a support 180.

The e-Petri dish 620 of these embodiments is an on-chip design where thespecimen 150 can be stored and in some cases imaged. This on-chip designmay be suitable to replace conventional petridishes and well-plates inbiology labs. The e-Petri dish 620 of embodiments may include anysuitable combination of structures and devices for holding the specimen150, maintaining the environment of the specimen 150, and/or imaging thespecimen 150. For example, an e-Petri dish 620 may include a chamberdesign (e.g., well 170), which can be placed on an imaging sensor chipwhere cells and the culture medium can be stored. The chamber design caninclude an array of chambers. As another example, an e-Petri dish 620may also include a fluidic network with one or more fluid channels. Thefluidic network can also be designed to provide control of chemical andmechanical environment in the e-Petri dish 620. As another example, ane-Petri dish 620 may also include one or more dielectric cages forholding the specimen 150 or object(s) 152 in the specimen 150 in an areasuch as the sensing area 164. As another example, an e-Petri dish 620may also include a support 180 for holding the e-Petri dish 620. InFIGS. 11, 13(a), and 13(b), the e-Petri dish 620 is comprised of a lightdetector 160 having a sensing surface 162, a transparent layer 165having a specimen surface 140, and a well 170, and a cover 176.

The transparent layer 165 of the e-Petri dish 620 of embodiments may beany suitable material layer capable of separating the specimen 150 fromthe light sensitive region of the light detector 160. The transparentlayer 165 may be a part of the light detector 160 or may be a separatelayer (e.g., coating) lying over the light detector 160. The transparentlayer 165 includes a specimen surface 140 for receiving the specimen150. The transparent layer 165 may be made of any suitable material suchas Polydimethylsiloxane (PDMS). The transparent layer 165 may have anysuitable thickness (e.g., thickness in the range of several hundrednanometers to microns). In one example, the transparent layer 165 is 0.9μm thick. In an embodiment where the transparent layer 165 is a layerlying over the light detector 160, the transparent layer 165 may be apassivation layer coated or deposited on top an imaging sensor chip. Thetransparent layer 165 may be comprised of multiple layers of differenttransparent materials in some cases. For example, the transparent layer165 may be comprised of a thin passivation layer, a coating, and/or aculture medium.

The e-Petri dish 620 also includes a light detector 160 in the form ofan imaging sensor chip capable of generating one or more signals withlight data associated with the projection images 160 captured and otherdata associated with imaging. The light detector 160 can be amonochromatic detector or a color detector (e.g., RGB detector).Suitable imaging sensor chips are commercially available. In some cases,the light detector 160 includes a two-dimensional array of discretelight detecting elements 166 (shown in FIG. 2). Some examples ofsuitable light detectors 160 that have two-dimensional arrays ofdiscrete light detecting elements 166 include a charge coupled device(CCD) array, a CMOS imaging sensor array, an avalanche photo-diode (APD)array, a photo-diode (PD) array, and a photomultiplier tubes (PMT)array. The arrays of light detecting elements can have any suitableorientation. Also, the light detecting elements 166 may be of anysuitable size (e.g., 1-10 microns) and any suitable shape (e.g.,circular, rectangular, square, etc.). For example, a CMOS or CCD lightdetecting element 166 may be 1-10 microns and an APD or PMT lightdetecting element 166 may be as large as 1-4 mm. The light detector 160also includes a sensing surface 162 that has a sensing area 164, whichcan refer to an area of the sensing surface 162 that actively capturesimage projections 170. Although illustrated embodiments show the sensingarea 164 covering a small portion of the sensing surface 162, in otherembodiments, the sensing area may cover a larger portion or the entiresensing surface 162.

In some cases, the transparent layer 165 may be placed on the sensingsurface 162 of an imaging sensor chip during fabrication of the chipusing suitable fabrication procedures such as semiconductor and/ormicro/nanofabrication procedures. In one case, the transparent layer 165may be prepared by mixing 1:10 with base and curing agent, then spincoated on a 3 in. silicon wafer followed by baking at 80 degrees C. Dueto the scattering angle of light 118 passing through the specimen 150,projection image quality can be degraded if the specimen 150 is locatedaway from the sensing surface 162. For this reason, the transparentlayer 160 may be a thin layer. Also, layers (e.g., color filter and amicrolens layers) of the imaging sensor chip may be removed or omittedto decrease the acceptance angle of each light detecting element (pixel)and the distance between the specimen 152 and the sensing surface 162.To remove the color filter and a microlens layers from a pre-fabricatedimaging sensor chip, the chip may be treated under oxygen plasma for aperiod of time (e.g., 10 minutes at 80 W).

In embodiments such as FIGS. 11 and 13(a), the e-Petri dish 620 alsoincludes a well. The well 170 can refer to any suitable structure orstructures capable of containing or holding a specimen 150 being imaged.The components of the well 170 such as the peripheral wall 172 may bemade of any suitable material such as PDMS. In embodiments such as shownin FIG. 11, the well 170 includes a peripheral wall 172 directly coupledto the specimen surface 140. The peripheral wall 172 forms a squareenclosure for containing the specimen 150. In other embodiments such asthe embodiment shown in FIG. 13(a), the well 170 may be separatestructure with a peripheral wall 172 and a transparent floor attached ata peripheral edge of the peripheral wall 172. The transparent floor maybe coupled, removably or otherwise, to the specimen surface 140 or tothe sensing surface 162 of the light detector 160. In other embodiments,the well 170 may include other structures such as an array of chambers.Each chamber capable of containing its own specimen 150.

In embodiments such as FIGS. 11 and 13(a), the e-Petri dish 620 alsoincludes a cover 176 that can be placed over the well 170. The cover 176may be any suitable structure that can provide suitable protection tothe specimen 150. The cover 176 may be made of any suitable material(e.g., PDMS) and may have any suitable thickness (e.g., 100 μm). Forexample, a cover 176 may be a thin sheet of PDMS that preventsevaporation of the culture media while allowing for CO₂ exchange betweenthe well 170 and the exterior. The cover 176 may be removable in somecases.

In embodiments such as FIGS. 11, 12, 13(b) and 13(c), the e-Petri device610 may include a support 180. A support 180 can refer to suitablestructures, devices or combination thereof capable of holding thescanning illumination source 110 in a fixed position relative to thelight detector 160 and at a distance, d, from the specimen surface 140.The support 180 may be part of the e-Petri dish 620 in some cases. InFIG. 11 and FIG. 13(b), the support includes a bottom portion with anopening for receiving and/or coupling to the e-Petri dish 620. Thesupport 180 of embodiments, such as shown in FIG. 13(b), also includes aportion for receiving the scanning illumination source 110. In somecases, such as the illustrated example of FIG. 11, FIGS. 13(b), and13(c), the scanning illumination source 110 may be held such that thedisplay surface 119 is kept approximately parallel to the sensingsurface 162 of the light detector 160 and at a distance, d, from thesensing surface 162 during scanning. In these cases, the illuminatingdisplay 116 may provide illumination 118 at angles normal to the displaysurface 119. In other cases, the scanning illumination source 110 may beheld so that the display surface 119 may be tilted at an angle fromnormal. At this angle, projections 170 from more extreme illuminationangles to be captured, leading to a more complete 3D reconstruction insome cases. In one embodiment, the scanning illumination source 110 mayinclude actuator(s) and controller(s) or other mechanism to repositionthe illuminating display 116 (e.g., LCD array) at an angle from normal.

In embodiments, a specimen 150 located on the specimen surface 140 maybe imaged by the e-Petri device 610 or the SMLM device 100. The specimen150 may include any suitable number (e.g., 1, 2, 10, 100, 1000, etc.) ofobjects 152 or portions (e.g., cell components) of objects 152. Thespecimen 150 may also include other material such as a culture medium.In FIG. 11, a specimen 150 with five objects 152 (e.g., cells) islocated on the specimen surface 140 within the wall 172 of the well 170.Any suitable specimen 150 may be imaged by the e-Petri system 600 ore-Petri device 610. For example, a suitable specimen 150 may be aconfluent sample (e.g., cell culture) having one or more objects (e.g.,cells). As another example, a suitable specimen 150 may be a sample inwhich the objects 152 are contiguously connected. The specimen 150 mayinclude any suitable type of object 150. Suitable types of objects 150can be biological or inorganic entities. Examples of biological entitiesinclude whole cells, cell components, microorganisms such as bacteria orviruses, cell components such as proteins, etc. Inorganic entities mayalso be imaged by embodiments of the invention.

In embodiments such as FIGS. 11, 12, 13(b) and 13(c), the e-Petri device610 includes a scanning illumination source 110. The scanningillumination source 110 may include any suitable device or combinationof devices capable of providing illumination 118 from differentillumination angles to a specimen 150 located on the specimen surface140 to generate sub-pixel shifted projections 170 of the specimen 150 atthe sensing surface 162. Suitable scanning illumination sources 110 arecommercially available. For example, a scanning illumination source maybe a mobile communication device (e.g., smartphone, tablet, etc.) havingan illuminating display 116. Illustrated examples of a suitable scanningillumination device 110 in the form of a smartphone are shown in FIGS.2, 4, 12, 13(b), and 13(d). Another example of a suitable scanningillumination device 110 may be a tomographic phase microscope that usesa spatial light modulator to scan illumination 118. In illustratedembodiments, the scanning illumination source 110 is in the form of amobile communication device.

In FIG. 11, the scanning illumination source 110 includes a firstprocessor 112, a first CRM 114, and an illuminating display 116. Theilluminating display 116 includes a display surface 119 and a lightelement 117 providing illumination 118 at the display surface 119.

The illuminating display 116 may be any suitable display capable ofproviding illumination 118. Suitable illuminating displays 116 arecommercially available. Some examples of suitable illuminating displays116 include monochromatic, color, or gray-scale LCDs, LED displays(e.g., display panels), television screens, LCD matrixes, etc. Theilluminating display 116 may be in the form of a two-dimensional arrayof light emitting components (e.g., light pixels) with a dimension M×Nof any suitable value (e.g., 1000×1000, 1000×4000, 3000×5000 etc.). Eachlight emitting component in the two-dimensional array may have alocation denoted as (x_(i), y_(j)) where i=1 . . . M; and j=1 . . . N.The illuminating display 116 includes a display surface 119. Theilluminating display 116 may be in any suitable position to provideillumination 118. In FIG. 11, the illuminating display 116 is positionedso that the display surface 119 is parallel to the sensing surface 162and is at a distance, d, from the sensing surface 162.

A light element 117 can refer to any suitable device capable ofproviding illumination 118. In FIG. 11, the light element 117 is shownlocated at the display surface 119. In other embodiments, the lightelement 117 may be located below the display surface 119. Inembodiments, the light element 117 can be a set of one or moreilluminated light emitting elements (e.g., LCD lit/pixel) of theilluminating display 116 at a given illumination time during anillumination cycle. The set may have any suitable number (e.g., 1, 5,10, 100, etc.) of light emitting components. In these embodiments, thelight element 117 may be different sets of light emitting elementsilluminated at different times during the illumination cycle. The lightelement 117 may be any suitable size and shape (e.g., rectangle, circle,spot, bar, etc.). In FIG. 4(a), the light element 117 has the shape of acircular spot comprising a set of 640 illuminated pixels of about 1 cmin diameter on the illuminating display 116. In addition to theposition, the size and shape of the different sets of illuminated lightemitting elements may vary over time. In other embodiments, the lightelement size and shape may be constant.

During an illumination cycle, the scanning illumination source 110 mayprovide the light element 117 at a plurality of positions at differenttimes to provide illumination 118 to the specimen 150 from a pluralityof illumination angles. For example, the light element 117 may bedifferent sets of light emitting elements at a plurality of positions onan illumination display 116. The scanning illumination source 110 maychange the light element 117 to the plurality of different positions byilluminating different sets of light emitting elements at differenttimes. In this example, the position of each light element 117 can referto the coordinates of the center of the set of illuminated lightemitting components. As another example, the light element 117 may betilted at a plurality of different angles at different times. As anotherexample, the light element 117 may be a single light source that ismoved to the plurality of positions. As another example, the lightelement 117 may be different sets of light sources at the plurality ofpositions in the incubator 800 that are illuminated at different times.The plurality of positions may include any suitable number, n, ofpositions (n=1, 2, 3, 4, 5, 10, 20, 100, 1000, etc.). The plurality ofillumination angles may include suitable illumination angles that cangenerate sub-pixel shifted projections 170 of the specimen 150 at thesensing surface 162. In one case, the light element 117 may bepositioned to generate a small range of illumination angles (e.g., +/−2degrees) in X/Y around a normal to the sensing surface 162.

In FIGS. 11, 12, 13(b), and 13(c), different sets of one or more lightemitting components (e.g., pixels) of the illuminating display 116 maybe illuminated at different times to change the position of the lightelement 117 and/or properties of the illumination 118 from the lightelement 117. In these illustrated examples, the scanning illuminationsource 110 is in the form of a smartphone having an illuminating display116 in the form of a two-dimensional array of light emitting components.Different sets of light emitting components in the two-dimensional arraymay be illuminated at different times during the illumination cycle.

In embodiments, the scanning illumination source 110 may position thelight element 117 in any suitable plurality of positions and the lightelement 117 may have any suitable properties during the illuminationcycle. In some cases, the plurality of positions may as a group form apattern (e.g., array, circle, square, triangle, etc.). In oneembodiment, the light emitting components of an illuminating display 116may be illuminated during an illuminating cycle according to a scanningpattern. A scanning pattern can refer to a description that includes theplurality of positions of the light element 117 at differentillumination times during an illumination cycle and the properties(e.g., size, shape, etc.) of the light element 117 at each position. Inone embodiment, a scanning pattern may be in the form of atwo-dimensional array (n×m dimensions) of positions of the light element117 at (x_(i=1 to n), y_(j=1 to m)) of the illuminating display 116. Thearray may have any suitable dimension (e.g. 1×100, 1×10, 100×100,3000×20, 400×300 etc.). In one example, a scanning pattern may include atwo-dimensional array of scanning locations and a description that thelight element 117 moves through each row sequentially at a constantrate. In another example, the scanning pattern may include atwo-dimensional array of scanning locations and a description that theelement moves through each column sequentially at a constant rate. Asanother example, the scanning pattern may include a two-dimensionalarray of scanning locations and a description that the element movesthrough the array randomly. The scanning pattern may also include theamount of sub-pixel shift desired between subsequent projection images.The scanning pattern may also include the total number of projectionimages and/or HR images desired.

FIG. 4(b) illustrates an example of a scanning pattern of anillumination cycle according to an embodiment. The scanning pattern isin the form of a graph of a 15×15 two-dimensional array of 225 positionsof the light element 117 on the display surface 119 during anillumination cycle. The 225 positions are shown in terms of x and ylocations along the x-axis and y-axis in the plane of the displaysurface 119 of the illuminating display 116. In the illustrated example,the scanning pattern includes 15 columns of positions in the x-directionand 15 rows of positions in the y-direction. At each position, the lightdetector 160 may capture a different projection image. The lightdetector 160 may capture as many as 225 different projection imagesbased on the 225 different scanning positions in the scanning pattern.The projection images captured by the light detector 160 may compriseone or more sequences of sub-pixel shifted projection images. The arrowsin the scanning pattern FIG. 4(b) designate the sequence in time of thepositions during the illumination cycle. According to the illustratedarrows, the light element 117 moves sequentially through each row of thetwo-dimensional array in the scanning pattern.

The scanning pattern may be stored as code on the first CRM 114 or thesecond CRM 220 and executed by the first processor 112 or secondprocessor 210. For example, the scanning pattern may be a video program(e.g., app on a smartphone) of a suitable format stored on the first CRM114 that when executed displays a video of the light element 117 movingto different positions on the illuminated display 116 over time. Anexample of such a video is illustrated in FIG. 4(a). In this example,the light element 117 is in the form of a light spot moving across theilluminating display 116 over time according to positions defined in thescanning pattern in FIG. 4(b). In FIG. 4(a), the light element 117 is atone position in the scanning pattern shown in FIG. 4(b). In thisexample, the illuminating display 116 is centered over the e-Petri dish620. The light element 117 can have constant or varied properties atdifferent positions. In one case, the light element 117 may remain aconstant size as it moves away from the center of the illuminatingdisplay 116. In this case, the intensity readout from the light detector160 associated with the specimen 150 will decrease away from the centerbecause of the large incident angle. In the illustrated case shown inFIGS. 4(a) and 4(b), to maintain a more constant intensity readout, thesize of the light element 117 (e.g., bright spot size) is linearlyincreased as it moves away from the center of the illuminating display116 (e.g., smartphone screen).

In embodiments, the properties (e.g., size, properties of theillumination 118, shape, etc.) of the light element 117 may vary atdifferent positions during an illumination cycle. The properties of thelight element 117 at different positions may be varied by changing thenumber of light emitting elements in the light element 117, the shape ofthe light element 117, and/or the properties of the light 118 from thelight emitting elements in the light element 117. The light properties(e.g., intensity, wavelength, frequency, polarization, phase, spinangular momentum and other light properties) of the illumination 118from a light element 117 at an illumination time during an illuminationcycle have any suitable values. The illumination 118 may be incoherentlight in some cases.

The light intensity requirement of the imaging schemes used by thee-Petri device are relatively low in embodiments. In one embodiment, asuitable intensity of illumination 118 can be the provided byillumination from a conventional smartphone screen. As a point ofreference, a halogen-lamp based conventional microscope typicallydelivers a light intensity of 20 W/m² onto a specimen 150. In otherembodiments, a suitable intensity of illumination 118 can be theprovided by a LED display panel, a television screen or a LED matrix. Asuitable light intensity received by a light detecting element 166 maybe 0.015 W/m².

In an embodiment, the intensity of the illumination 118 generated by thelight element 117 in an illuminating display 116 may be controlled byvarying the size of the light element 117. In some cases, the size ofthe light element 117 at different positions during an illuminationcycle may vary based on the distance between the position of the lightelement 117 and a point at the plane of the sensing surface 162 togenerate light of approximately the same intensity at that point. Inthis case, the size, S of the light element 117 at a position can beproportional to the distance, L, from the position to a suitablelocation of a point such as: a) the center of the array of scanninglocations, or b) the center of an illuminating display 116 such as thecenter of an LCD on a smartphone. For example, the size, S of the lightelement 117 at a position in the illumination cycle may be defined as:S=S_(center)×(1+L), where S_(center) is the size of the light element117 at the center of the array of positions. In this way, the lightintensity received at the location at the sensing surface 162 normal tothe center of the positions of the light elements 117 on the displaysurface 119 may be kept approximately constant. As another example, thesize S of the light element 117 at any position of the light element 117may be defined as: S=S_(A)×(1+A), where S_(A) is the size of the lightelement 117 at a location A of an illuminating display 116, A is thedistance from the position to the location A.

In an embodiment, the light element 117 can provide illumination 118 ofn different wavelengths λ₁, . . . , λ_(n) at n different illuminationtimes during an illumination cycle. The illumination 118 may be cycledthrough a series of different wavelengths as the light element 117 movesthrough different positions in an illumination cycle in some examples.In one example, the light element 117 can provide RGB illumination ofthree wavelengths λ₁, λ₂, and λ₃ corresponding to red, green, bluecolors, respectively. The light element 117 may provide illumination 118of the three wavelengths λ₁, λ₂, and λ₃ sequentially during illuminationtimes of a illumination cycle. In one case, at a illumination time t₁illumination 118 may have a wavelength of λ₁, at t₂ illumination 118 mayhave an wavelength of λ₂, at t₃ illumination 118 may have a wavelengthof λ₃, at t₄ illumination 118 may have a wavelength of λ₁, at t₅illumination 118 may have a wavelength of λ₂, etc. In this embodiment,the light detector

During an illumination cycle, illumination 118 from the plurality ofillumination angles generates a plurality of light projections 170 onthe sensing surface 162. Each projection image (frame) can refer to asnapshot image sampled by the light detector 160 at a sampling timeduring an illumination cycle. In some cases, the light detector 160 maycapture a projection image 170 at each illumination time. Eachprojection image sampled by the light detector 160 can be used todisplay a 2D projection image. In embodiments with a color lightdetector 160, the 2D projection image may be a color image. Inembodiments with a monochromatic light detector 160, the projectionimage may be a black and white image.

A sequence of sub-pixel shifted projection images can refer to nprojection images sampled at n sampling times where neighboring (in timeor space) projection images are separated by less than a pixel size(i.e. sub-pixel shift). During an illumination cycle, n projectionimages (I₁, . . . , I_(n)) may be captured at n sequential samplingtimes (t₁, . . . t_(n)). Any suitable number (e.g., 1, 3, 5, 10, 100,etc.), n, of projection images may be captured by the light detector 160during a illumination cycle. Also, any suitable number (e.g., 1, 3, 5,10, 100, etc.) of sequences of sub-pixel shifted projection images maybe captured by the light detector 160 during a illumination cycle. Ifmultiple sequences are captured, the sequences can include differentgroups of projection images or the sequences can overlap sharing one ormore projection images. In one example, 9 projection images (I₁, I₂, I₃,I₄, I₅, I₆, I₇, I₈, I₉) may be captured at 9 sequential sampling times(t₁, t₂, t₃, t₄, t₅, t₆, t₇, t₈, t₉). In an overlapping case of theabove example, sequences could be: 1) I₁, I₂, I₆, and I₈, and, 2) I₆,I₇, I₈, and I₉. In a non-overlapping case, sequences could be: 1) I₁,I₂, I₃, and I₄, and 2) I₅, I₆, I₇, and I₈. In others examples, asequence of sub-pixel shifted projection images may be based onnon-sequential sampling times. For example, 9 projection images (I₁, I₂,I₃, I₄, I₅, I₆, I₇, I₈, I₉) may be captured at 9 sequential samplingtimes (t₁, t₂, t₃, t₄, t₅, t₆, t₇, t₈, t₉) and the sequence ofprojection images may be (I₆, I₂, I₉, I₁).

In embodiments, the light detector 160 may capture a projection image ateach position of the light element 117 in a scanning pattern. Forexample, a light detector 160 may capture 225 projection imagesassociated with the 15×15 array of positions in the scanning patternshown in FIG. 4(b). In this example, the light detector 160 may capturea projection image at each position as the light element 117 movesthrough each row sequentially of the two-dimensional array of positionsin the scanning pattern. If the positions in each row are associatedwith 15 sub-pixel shifted projections 170, the light detector 160 maycapture 15 sequences of 15 sub-pixel shifted projection images duringeach illumination cycle. In this case, each of the 15 sequences capturedis associated with a row of positions of the light element 117 in thescanning pattern.

In FIG. 11, the first processor 112 of the illumination source 110 is inelectronic communication with the illuminating display 116, the firstCRM 114, and the light detector 160. The first processor 112 (e.g.,microprocessor) can execute code stored on the first CRM 114 (e.g.,memory) to perform some of the functions of the scanning illuminationsource 110. For example, the first processor 112 may execute code with ascanning pattern stored on the first CRM 114. The CRM 114 may include,for example, code with a scanning pattern, other code for scanning alight element 117, and other codes for other functions of the scanningillumination source 110. The first CRM 114 may also include code forperforming any of the signal processing or other software-relatedfunctions that may be created by those of ordinary skill in the art. Thecode may be in any suitable programming language including C, C++,Pascal, etc.

Light data can refer to any suitable information related to the one ormore projections 170 captured by the light detecting elements 166 of thelight detector 160. For example, light data may include informationabout the properties of the projection light received such as theintensity(ies) of the light, the wavelength(s) of the light, thefrequency or frequencies of the light, the polarization(s) of the light,the phase(s) of the light, the spin angular momentum(s) of the light,and/or other light properties associated with the light received by thelight detecting element 166. Light data may also include the location ofthe receiving light detecting element(s) 166, the time that the lightwas received (sampling time or scanning time), or other informationrelated to the projection 170 received. In embodiments, each lightdetecting element 166 can generate a signal with light data based onlight associated with the projection 170 and received by the lightdetecting element 166.

A motion vector can refer to the translational motion of projectionimages in a sequence of projection images, collectively termed themotion vector of the sequence of projection images. The motion vector isbased on the amount of shifting of the projection images at a plane. Amotion vector of a sequence of sub-pixel shifted projection images canbe calculated from the associated projection images captured by thelight detector 160. The motion vector may be calculated at any plane ofinterest. For example, the motion vector can be determined at the planeat the sensing surface 162. In this example, the motion vector isdetermined in terms of the local x′-axis and y′-axis at the sensingsurface 162 of the light detector 160. As another example, the motionvector can be calculated at other planes through the specimen 150 beingimaged. The planes through the specimen 150 may be parallel to the planeof the sensing surface 162 in some cases.

In embodiments of the e-Petri system 600, a sub-pixel resolution imageof a specimen 150 can be constructed using a suitable SR algorithm basedon data associated with a sequence of sub-pixel shifted projectionimages and a motion vector of the sub-pixel shifted projections in thesequence. An example of image resolution obtainable by embodiments ofthe e-Petri system 600 may be about 0.66 micron. Any suitable SRalgorithm can be used. An example of a suitable SR algorithm is ashift-and-add pixel SR algorithm. Other examples of SR algorithms arediscussed in Section II.

A sub-pixel resolution image generated based on a motion vector will befocused at a plane of interest associated with the motion vector. Thatis, if a motion vector is estimated based on a plane of interest, thesub-pixel resolution image will be a two-dimensional image focused atthe plane of the interest used to estimate the motion vector. Forexample, if a motion vector is estimated based on a plane of interest atthe sensing surface 162, the sub-pixel resolution image generated willbe focused at the plane of the sensing surface 162. If the motion vectoris estimated based on a plane through the specimen 150 being imaged, thesub-pixel resolution image will be a cross-sectional image of thespecimen 150 focused at the plane through the specimen 150. In anembodiment, an e-Petri system 600 can generate a sub-pixel resolutionimage of a cross-section of the specimen 150 by modifying the value ofthe motion vector used to generate the sub-pixel resolution image to theplane at the cross-section. The e-Petri system 600 can vary the valuesof the motion vector to focus at various cross sections of the specimen150. In an embodiment, an e-Petri system 600 can generate athree-dimensional sub-pixel resolution image based on multipletwo-dimensional cross-sectional sub-pixel resolution images generatedusing multiple motion vector values associated with multiple planesthrough the specimen 150.

The e-Petri system 600 of FIG. 11 also includes a host computer 200communicatively coupled to the light detector 160. The host computer 200comprises a second processor 210 (e.g., microprocessor), a second CRM220, and an image display 230. The image display 230 and the second CRM220 are communicatively coupled to the second processor 210.Alternatively, the host computer 200 can be a separate device from thee-Petri system 600. The host computer 200 can be any suitable computingdevice (e.g., smartphone, laptop, tablet, etc.).

The second processor 230 executes code stored on the second CRM 220 toperform some of the functions of the e-Petri system 600 such as, forexample: interpreting data from one or more sequences of sub-pixelshifted projection images captured and communicated in one or moresignals from the light detector 160, determining a motion vector of asequence of sub-pixel shifted projections, constructing a 2D HR imagefrom data associated with a sequence of sub-pixel shifted projectionimages, constructing a 3D HR image from data associated with a sequenceof sub-pixel shifted projection images, displaying one or more HR imageson the image display 230, automatedly reconstruct images and display thereconstructed images on the display 230 for user monitoring, etc.

The second processor 210 can receive one or more signals with light dataand other data from the light detector 160. For example, the processor210 can receive one or more signals with light data associated with oneor more sequences of sub-pixel shifted projection images sampled at acorresponding sequence of n illumination times (t₁, t₂, t₃, . . .t_(n)). The second processor 210 can also determine a motion vectorbased on the sequence of sub-pixel shifted projection images. The secondprocessor 210 can also construct HR images and associated image databased the determined motion vector and data associated with at least onesequence of sub-pixel shifted projection images. In some cases, theconstructed HR image of the object 150 is a black and white 2D/3D image.In other cases, the constructed HR image of the object 150 is a color2D/3D image.

In one embodiment, a HR color image can be generated by using differentwavelengths of illumination 118 at different sampling times to generatea multiple sequences of sub-pixel shifted projection images at a lightdetector 160. Each sequence is associated with a different wavelength.The second processor 210 can generate HR color image and associatedimage data based on the different sequences associated with differentwavelengths. For example, three wavelengths of light (e.g., wavelengthsassociated with red, green, blue (RGB) colors) can be sequentiallygenerated by a light element 117 to generate three sequences ofsub-pixel shifted projection images associated with three wavelengths oflight. The processor 210 can combine the image data from the sequencesassociated with the different wavelengths to generate multi-wavelengthor color image data (e.g., RGB color image data). The multi-wavelengthor color HR image data can be used to generate a multi-wavelength orcolor HR image on the image display 230.

Second CRM (e.g., memory) 220 can store code for performing somefunctions of the e-Petri system 600. The code is executable by thesecond processor 210. For example, the second CRM 220 of embodiments mayinclude: a) code with a SR algorithm, b) code with a tomographyalgorithm, c) code for interpreting light data received in one or moresignals from the light detector 160, d) code for generating a 3D HRimage, e) code for constructing a color sub-pixel image, f) code fordisplaying SR two-dimensional and/or three-dimensional images, g) codefor a customized program to automatedly reconstruct and display thereconstructed image onto the display 230 for user monitoring; and h)and/or any other suitable code for performing functions of the e-Petrisystem 600. The second CRM 220 may also include code for performing anyof the signal processing or other software-related functions that may becreated by those of ordinary skill in the art. The code may be in anysuitable programming language including C, C++, Pascal, etc.

The e-Petri system 600 also includes an image display 230communicatively to the processor 210 to receive data and provide outputsuch as HR images to a user of the e-Petri system 600. Any suitabledisplay may be used. For example, the image display 230 may be a colordisplay or a black and white display. In addition, the image display 230may be a two-dimensional display or a three-dimensional display. In oneembodiment, the image display 230 may be capable of displaying multipleviews of an object 150.

The e-Petri system 600 of embodiments also includes an incubator 800. Anincubator can refer to any suitable device/structure or combination ofdevices and structures that can provide a pre-defined environment at thee-Petri dish 620 during a time interval of the experiment (e.g., longterm study) being performed by the e-Petri system 600. The pre-definedenvironment may define environmental variables such as temperature,humidity, etc. Suitable incubators are commercially available. In somecases such as the illustrated example in FIGS. 11 and 12, the incubator800 may include a chamber defined by a wall. The chamber may be designedto hold one or more e-Petri-devices 610.

The e-Petri system 600 of embodiments also includes a relay 700. A relay700 can refer to a suitable device that can relay data from the one ormore e-Petri devices 610 to the host computer 200. Suitable devices arecommercially available. The relay 700 may include multiplexingfunctionality to combine signals from the one or more e-Petri devices610 to a signal to the host computer 200. The relay 700 may also includedemultiplexing functionality to extract data from a signal from the hostcomputer 200 to send in a signal to one or more e-Petri devices 610. Therelay 700 may transmit data wirelessly.

Modifications, additions, or omissions may be made to e-Petri system600, e-Petri device 610 or e-Petri dish 620 without departing from thescope of the disclosure. For example, an e-Petri system 600 of otherembodiments may omit the incubator 800 and/or the relay 700. As anotherexample, an e-Petri device 610 of other embodiments may omit the support180 or may have a single support 180 for multiple e-Petri devices 610.As another example, an e-Petri system 600 may omit the host computer200. As another example, an e-Petri system 610 may omit the well 170 andwall 172.

In addition, components of the to e-Petri system 600, e-Petri device 610or e-Petri dish 620 may be integrated or separated according toparticular needs. For example, an e-Petri system 600 may have a singlescanning illumination source 110 that provides illumination 118 tomultiple e-Petri devices 610. As another example, the relay 700 may belocated outside the incubator 800. As another example, the secondprocessor 610 may be integrated into the light detector 160 so that thelight detector 160 performs one or more of the functions of the secondprocessor 160 in another e-Petri system 600. As another example, thesecond processor 160, second CRM 220, and image display 230 may becomponents of a computer separate from an e-Petri system 600 and incommunication with the e-Petri system 600. As another example, thesecond processor 160, second CRM 220, and/or image display 230 may beintegrated into parts of the e-Petri device 610. For example, the imagedisplay 230 may be part of the illumination display 116, the firstprocessor 112 and second processor 210 may be integrated into a singleprocessor, and/or the first CRM 114 and second CRM 220 may be integratedinto a single CRM. As another example, more than one e-Petri dish 620and/or the incubator 800 can be incorporated into the e-Petri device610.

B. Wide Field-of-View Imaging Capabilities

The e-Petri device 610 of embodiments are capable of wide field-of-viewimaging. By providing a wide field of view, the e-Petri device 610 ofembodiments may be suitable to replace or improve upon conventionalmicroscopes for cell culture analysis.

To demonstrate these capabilities, a specimen 150 having HeLa cells wascultured for 48 hours on a specimen surface 140 of an e-Petri-dish 620of an e-Petri device 600 of an embodiment as shown in FIG. 13(c). Topromote cell adhesion, the specimen surface 162 of the e-Petri dish 620was treated with Poly-L-lysine (0.01%) for 15 min and washed 3 timeswith distilled water. The HeLa cells were first cultured in Dulbecco'smodified eagle medium supplemented with 1-glutamine (4 mM), penicillin(100 units/ml), streptomycin (100 μg/ml) and 10% (v/v) fetal calf serumin culture dishes and maintained in 5% CO2 humidified atmosphere at 37°C. During the logarithmic growth period, the HeLa cells were harvestedby trypsin (0.05% trypsin with EDTA*4Na), re-suspended in DMEM, and thenseeded onto the ePetri dish 620.

The specimen 150 was then stained with Giemsa, which is a blackprecipitate formed from the addition of aqueous solutions of methyleneblue and eosin, dissolved in methanol. The following steps were used tostain the cell culture: 1) fix the air-dried sample in absolute methanolby dipping the ePetri dish 620 briefly (two dips) in a Coplin jarcontaining absolute methanol; 2) remove and let air dry; 3) stain withdiluted Giemsa stain (1:20, vol/vol) for 20 min.; and 4) wash by brieflydipping the ePetri dish 620 in and out of a Coplin jar of distilledwater (one or two dips).

The e-Petri device 600 in FIG. 13(c) has an illumination source 110 inthe form of a smartphone with an illuminating display 116 in the form ofan LED display. During an exemplary illumination cycle, a video with alight element 117 in the form of a light spot was displayed on the LEDdisplay at a plurality of positions according to the 15×15 arrayscanning pattern shown in FIG. 4(b). FIG. 4(a) is illustrated example ofthe light spot 117 at one position. The light element 117 providedillumination 118 of three wavelengths λ₁, λ₂, and λ₃ corresponding tored, green, blue colors at different positions during the video. Theimaging run included a single illumination cycle. The sampling rate(i.e. image capture rate) of the light detector 160 was set to capture10 frames per second with a pixel clock of the light detector 160running at 70 Mhz. The entire data acquisition process of the imagingrun took about 20 seconds.

FIG. 7(a) shows the reconstructed color projection image of theconfluent HeLa cell sample. To illustrate the amount of detail in theconstructed color image shown in FIG. 7(a), reconstructed images ofselected regions of the HeLa sample are provided in FIGS. 7(b 2) and 7(c2). FIGS. 7(b 1) and 7(c 1) are projection images of the selectedregions of FIGS. 7(b 2) and 7(c 2). The image enhancement factor used inthe algorithm to generate the reconstructed image of FIG. 7(a) was setat 13. In other words, each pixel at the low-resolution projection imagelevel (2.2 μm) was enhanced into a 13×13 pixel block in thereconstructed image. The entire reconstructed image of FIG. 7(a)contains about 8.45×108 pixels. The e-Petri device 600 of FIGS. 13(b)and 13(c) took about 22 second to capture each sequence of projectionimages for each color. The solution for the reconstructed image wasnon-iterative, deterministic and was optimized in the Maximum-Likelihoodsense. With the use of a graphics processing unit (GPU), imageprocessing time may be less than one second for the entire imagereconstruction. Since the primary use of ePetri device 610 would be fortracking cell culture growth directly from within an incubator, the datatransfer or the processing speed limitations should not reduceefficiency of the system.

From the reconstructed high resolution color images in FIGS. 7(b 2) and7(c 2), organelles within the HeLa cell sample can be discerned such asthe multiple nuclear granules (indicated by red arrows), and thenucleus. FIG. 7(d) is a conventional microscopy image of similar cellsusing a microscope with 40×, NA=0.66 objective lens for comparison.

In an experiment, a HeLa cell sample was cultured on a CMOS sensor chipand fixed and stained with Giemsa. FIG. 14 (a 1), FIG. 14(a 2), and FIG.14(a 3) are conventional microscopy images with red, green and blue LEDilluminations (20× objective, 0.5 N.A.). FIG. 14 (a 4) is the colorimage constructed based on the red, green, and blue images in FIG. 14 (a1), FIG. 14(a 2), and FIG. 14(a 3). Since the sensor chip is nottransparent, the conventional microscopy images were taken in reflectionmode. The color in FIG. 14 (a 4) is due to the light interferencebetween the sensor surface and sample. The grid pattern in FIG. 14 (a 1)to FIG. 14 (a 4) is the pixel array of the image sensor (2.2 μm pixelsize). FIG. 14 (b 1), FIG. 14(b 2), and FIG. 14(b 3) are reconstructedsub-pixel resolution images of a portion of HeLa cell sample as acquiredby an e-Petri system 600 under red, green, and blue light sourcescanning respectively, according to an embodiment of the invention. FIG.14 (b 4) is the reconstructed sub-pixel resolution color image based onthe red, green, and blue images in FIG. 14 (b 1), FIG. 14(b 2), and FIG.14(b 3), according to an embodiment of the invention. The scale barshown in each of the images in FIG. 14 is 20 μm.

C. Long-Term Cell Imaging and Tracking

The e-Petri system 600 of embodiments can be automated to imageconfluent cell samples with sub-cellular resolution over a largefield-of-view at intervals of a study or other experiment. As such, itis well suited to long-term cell culture imaging and trackingapplications.

To demonstrate these capabilities, an e-Petri-system 600 such as shownin FIGS. 11 and 12 was used to perform a longitudinal cell imaging andstudy. A customized program stored on the second CRM 220 was used by thesecond processor 210 of the host computer 200 to automatedly reconstructand display the reconstructed image onto the display 230 for usermonitoring. The customized program included information such as the timeperiod of the experiment, the time intervals for automatedlyreconstructing and displaying the images, and other suitableinformation.

The e-Petri system 600 had an e-Petri device 610 such as shown in FIGS.12 and 13(c). A specimen 150 having HeLa cells was seeded onto aspecimen surface 140 of an e-Petri-dish 620 of an e-Petri device 600 ofan embodiment as shown in FIG. 13(c). The e-Petri device 610 was placedinto an incubator 800 such as shown in FIG. 12 during the longitudinalcell imaging and study. An Ethernet cable connected the e-Petri device610 to the host computer 200 in the form of a personal computer locatedthe incubator 800 for data transfer, as shown in FIG. 12. In thisexperiment, an automated reconstruction and displaying imaging run wasperformed at 15 minute intervals during the entire growth duration of 48hours. Each imaging run included a single illumination cycle. During the48 hour time period of the study, the number of HeLa cells grew from 40+to hundreds.

FIG. 15(a) are time-lapse reconstructed images of a portion of the HeLacell sample from a specific sub-location, as acquired by an e-Petrisystem 600 at starting at times, t=10 hr, t=17.5 hr, t=25 hr and t=32.5hr during the time period of the study, according to an embodiment ofthe invention. Based on the time-lapse cell image data acquired by thee-Petri device 610, the e-Petri system 600 can detect and track eachindividual cell's movements in space and time, and generatecorresponding lineage trees (i.e. mother-daughter relationship andotherwise analyze the data. FIG. 15(b) are graphs 232 of the trackingtrajectories of three cell families annotated by a biologist and thelineage trees for these cell families that were processed by the secondprocessor 210 of the host computer 200 of an e-Petri system 600,according to an embodiment of the invention.

D. Resolution

FIG. 8(a) is a example of a HR image of a specimen 150 having 500 nmmicrospheres (Polysciences) that may be acquired by an e-Petri system600, according to an embodiment of the invention. The imaging processused to construct the image was identical the one used to reconstructthe images in FIG. 7. For a single 500 nm microsphere, the bright centerof the microsphere was clearly resolved as shown in FIG. 8(a), with thefull-width at half maximum (FWHM) of 690 nm. FIG. 8(b) is an example ofa reconstructed image of a magnified small feature of the stained HeLacell specimen 150 of FIG. 7 as acquired by an e-Petri system 600,according to an embodiment of the invention.

In some cases, microscopy resolution may be defined based on a givenmicroscope's ability to resolve two closely spaced feature points. Toestablish resolution of an e-Petri system 600 of an embodiment based onthis definition, two closely spaced microspheres were imaged by thee-Petri system 600. FIG. 8(a) shows the reconstructed images of twoclosely packed 500 nm microspheres with center-to-center distance of 660nm, as acquired by the e-Petri system 600. The data trace in FIG. 8(a)shows a valley between the two peaks and, thus, establishes that theresolution may be 660 nm or better in some embodiments. To furtherverify this point, FIG. 8(b) shows the magnified small feature of thestained HeLa cell specimen of FIG. 7 and the FWHM of this feature wasestimated to be about 710 nm. In one embodiment, the estimatedresolution may be reduced if the specimen 150 is placed at a substantialdistance above the sensor surface 160, such as described in Heng, X., etal, “Characterization of light collection through subwavelength aperturefrom a point source,” Optics express 14, pp. 10410-10425 (2006); andWang, Y. M., Zheng, G., and Yang, C., “Characterization of acceptanceangles of small circular apertures,” Optics Express 17, pp. 23903-23913(2009), which are hereby incorporated by reference in their entirety forall purposes.

E. Additional Advantages

Embodiments of the invention provide one or more technical advantages. Ageneral advantage of embodiments may be a lensless, sub-pixelresolution, wide field-of-view imaging device capable of imagingconfluent cell culture with high resolution and incoherent lightsources. Images acquired by embodiments may be closely comparable withthose obtained with a conventional microscope. In an embodiment, animage with a resolution of 660 nm was acquired. The imaging method canbe implemented on a smart e-Petri dish, which is capable of performinghigh resolution and autonomous imaging of cells plated on or growing ona low-cost CMOS sensor chip. The e-Petri dish of embodiments may be auseful tool for in-vitro long-term cell observations or other long termstudies. To demonstrate that an e-Petri device 610 of an embodiment canbe easily assembled, the e-Petri device was constructed out of blocks, asmartphone, and an imaging sensor chip.

An advantage of an e-Petri dish 620, e-Petri device 610, and e-Petrisystem 600 may be low cost. The e-Petri dish 620 can use a CMOS imagingsensor chip as the base substrate for cell culture growth.Post-experiment, the sensor chip can either be disposed orwashed-and-reused. Given the low cost of these sensor chips, they areunlikely to represent a major cost component in most cell cultureexperiments.

Another of an e-Petri dish 620 may be that is can be disposable orrecyclable. In certain biohazardous experiments, the ability to treatthe sensor chips as disposable units would significantly reduce anyassociated risks.

Another advantage of an e-Petri system 600 may be that the host computer200 outside the incubator 800 can display direct readout from the dataacquired from inside the incubator 800. As shown in FIG. 12, the e-Petridevice 610 of embodiments may be sufficiently compact to fit comfortablyin a conventionally available incubator 800. In fact, given thefootprint of the e-Petri device 600 of embodiments, it may be possibleto fit multiple ePetri devices 610 into the same incubator 800, as shownin FIG. 11. Upon connecting the ePetri device 610 to an exteriorprocessor 210 via an appropriate data cable, a user can start to collectimages of the growing cell culture without removing the unit from theincubator 800. This advantage saves labor and cut down on theperturbations the cell culture is subjected to. In one embodiment, ane-Petri system 600 may include a compact and portable incubator 800 andePetri device 610 combination that is suitable for point-of-carediagnostic and/or other uses.

Another advantage of an e-Petri system 600 may be the ability tocontinuously monitor a specimen 150 from the incubator 800. A user of ane-Petri system 600 may be able to monitor cell growth continuously. Inbioscience research, this represents a good means for performinglongitudinal studies. In medical applications, this can significantlycut down on the diagnostic time for medical procedures that requiresculture growth based assessment. As an example, the ePetri dish 620 canreplace the standard petri dish for tuberculosis, staph and otherbacterial infection diagnosis. Whereas standard medical practice wouldstart initiate a bacteria culture growth and then check the growth atrelatively long time intervals (checking frequently would be too timeconsuming), a modified ePetri dish 620 may potentially be able tocontinuously and autonomously monitor for growth changes and notify theuser to examine the sample when significant changes have been detected.

Another of an e-Petri dish 620 may be that it is a platform technology.Since the top surface of the light detector 160 in the form of a sensorchip may be unmodified in embodiments, a user is free to build upon it.It is very possible to simply use the ePetri as an imaging platform fora large number of sophisticated lab-on-a-chip designs, such asmicroorganisms detection based on the use of closed dielectrophoreticcages, microfluidics-based phenotyping imaging and screening ofmulticellular organisms, and high throughput malaria infectederythrocyte separation and imaging. An example of suitable closeddielectrophoretic cages can be found in Medoro, G. et al. “Alab-on-a-chip for cell detection and illumination,” Optics letters 35,pp. 2188-2190 (2010), which is hereby incorporated by reference in itsentirety for all purposes. An example of microfluidics-based phenotypingimaging and screening of multicellular organisms can be found in Crane,M., Chung, K., Stirman, J. & Lu, H., “Microfluidics-enabled phenotyping,imaging, and screening of multicellular organisms,” Lab on a Chip 10,pp. 1509-1517 (2010), which is hereby incorporated by reference in itsentirety for all purposes. An example of high throughput malariainfected erythrocyte separation and imaging can be found in Hou, H., etal., “Deformability based cell margination—A simple microfluidic designfor malaria infected erythrocyte separation,” Lab on a Chip 10, pp.2605-2613 (2010), which is hereby incorporated by reference in itsentirety for all purposes. It is also possible to modify the ePetri dish620 to serve as a self-contained incubator and imaging unit. Also, tocreate a fluorescence imaging e-Petri dish 620, the light detector 160may be coated with an appropriate filter material.

F. Flowchart

The e-Petri system 600 of embodiments can be automated to reconstructand display sub-pixel resolution images 231 of a specimen 150 on ane-Petri dish 620 and other data 232 at intervals during an experimentfor user monitoring. For example, a cell specimen 150 may be placed onan e-Petri dish 620 having a culture medium. The e-Petri dish 620 maythen be placed into the support 180 of an e-Petri device 610 and thee-Petri device 610 with the e-Petri dish 620 may be placed in anincubator 180 for the duration of the experiment. The host computer 200can receive a signal with data associated with the sub-pixel shiftedprojection images captured by the e-Petri dish 620 and reconstructimages 231 and other data 232 on the host computer at intervals duringthe experiment for user monitoring without having to remove the e-Petridevice 610 from the incubator 800. The intervals may be at times T=T₀,T₁ . . . T_(n), where n can be any suitable integer (1, 2, 10, 100,etc.). T₀ refers to the initial time at the beginning of the experiment.

In some cases, a code may be used to automatedly reconstruct and displaythe images 231 and other data 232 at intervals. The code may be, forexample, a customized program stored on the second CRM 220. The secondprocessor 210 may use the code to automatedly reconstruct and display asub-pixel resolution image on the display 230 for user monitoring. Thecustomized program may include any suitable information regarding theimaging functions of the systems and the parameters. For example, thecustomized program may include the time intervals between illuminationcycles, the time period of the experiment, the scanning pattern usingduring the illumination cycles, etc.

FIG. 16 is a flow chart of an exemplary method of automatedlyreconstructing and displaying the sub-pixel resolution images 231 andother data 232 at intervals during an experiment by an e-Petri system600, according to embodiments of the invention. In step 910, a specimen150 is placed onto the specimen surface 140 of the e-Petri dish 620. Ifthe e-Petri dish includes a well 170 with a peripheral wall 172, thespecimen 150 is placed within the peripheral wall 172 of the well 170 onthe specimen surface 140. The e-Petri dish 620 may also have a culturemedium or other material located on the specimen surface 140 of thee-Petri dish 620. Once the specimen 150 is introduced, the e-Petri dish620 is then placed in the e-Petri device 600. For example, the e-Petridish 620 may be placed in a receiving portion of the support 180 of thee-Petri device 610.

In step 920, the e-Petri device 610 is located within an incubator 800providing a pre-defined environment. The e-Petri device 610 can remainwithin the incubator 800 during the duration of the experiment, whichmay reduce contamination risks. Once in the incubator, the e-Petrisystem 600 can automatedly image the specimen 150 at time intervalsT=T₀, T₁ . . . T_(n), where n can be any suitable integer (1, 2, 10,100, etc.). T₀ refers to the initial time at the beginning of theexperiment. The time intervals can be on a periodic basis such as every15 minutes. The experiment may have any suitable duration. The e-Petrisystem 600 may also generate other data 232 associated with the datagathered by the e-Petri dish 620. For example, various graphs of thegrowth of different cell lineages may be generated and displayed at eachinterval. The e-Petri system 600 may use a code stored on the CRM 220 orother suitable memory. The code include processing instructions that areautomated during the experiment. For example, the code may includedinstructions on reconstructing and displaying images 231 and other data232 at intervals. The code may also include instructions regarding theillumination cycle at each interval and other instructions regardingfunctions of the e-Petri system 600.

In step 930, the specimen 150 is illuminated from a plurality ofillumination angles at different times during an interval of theexperiment. The e-Petri device 610 includes a scanning illuminationsource 110 at a distance, d, from the sensing surface 162 that provideslight 118 from the plurality of illumination angles to the specimen 150on the specimen surface 142. Light 180 from the plurality ofillumination angles generates one or more sequences of sub-pixel shiftedprojection images on the sensing surface 162 of the light detector 160of the e-Petri dish 620.

In one embodiment, the illumination source 110 may have an illuminatingdisplay 116 with a light element 117 providing the illumination 118 fromdifferent angles at different times during an illumination cycle duringthe interval of the experiment. The light element 117 is positioned atdifferent positions at different times to provide the illumination 118from the different angles. For example, the light element 117 may bedifferent sets of illuminated light emitting elements of theilluminating display 116 at different times during the experiment. Theillumination 118 can vary at different positions. For example, the lightelement 117 can provide illumination 118 of n different wavelengths λ₁,. . . , λ_(n) at different times during each illumination cycle toobtain a sequence of projection images for each wavelength. Any suitablenumber of wavelengths may be used (e.g., n=1, 2, 3, 4, 5, . . . , 20).In one embodiment, the light element 117 may provide illumination 118 ofthree wavelengths λ₁, λ₂₃, and λ₃ corresponding to red, green, bluecolors at different sampling times. In some cases, the illumination 118from one scanning location to a neighboring scanning location may havedifferent wavelengths. In other cases, the illumination 118 may have afirst wavelength during a first series of scanning positions, and thenprovide illumination 118 of a second wavelength during a second seriesof scanning positions, and so forth until n sequences of projectionimages corresponding to n different wavelengths have been captured. Insome cases, the first processor 112 of the illumination source 110 maydetermine a scanning pattern having the different positions at thedifferent times, properties (e.g., wavelength(s) of light used, the sizeand shape of the light element 117, the intensity(ies) of the lightelement, etc.) of the light element 117 at different scanning positions,the amount of sub-pixel shift desired in a sequence, the duration of theexperiment, the interval time(s), descriptions of other data that may bedisplayed during each interval, and other suitable information relatedto the operation of the e-Petri system 600. The scanning pattern may bestored as code on the first CRM 114 or other suitable memory.

In step 940, the light detector 160 captures one or more sequences ofsub-pixel shifted projection images of the specimen 150 during eachillumination cycle of each interval during the experiment. The lightdetector 160 may capture a projection image at each illumination angle.

In step 950, the processor 230 uses a suitable method to determine themotion vector of the one or more sequences of sub-pixel shiftedprojection images captured during the interval of the experiment. Themotion vector is determined based on a plane of interest such as thesensing surface 162 or plane through the specimen 150. Any suitablemethod of determining a motion vector can be used.

In step 960, the processor 210 uses an appropriate SR algorithm toconstruct one or more sub-pixel images of the specimen from the one ormore sequences of sub-pixel shifted projection images and the determinedmotion vector during the interval of the experiment. The one or moresub-pixel resolution images are located (focused) at the plane ofinterest used to determine the motion vector. Depending on the scheme,the sub-pixel resolution images can be 2D monochromatic images, 2D colorimages, 3D monochromatic or color images. The processor 210 may alsogenerate other data images 232 (e.g., lineage trees) based on datagathered during the interval.

In step 970, the processor 210 can display the one or more sub-pixelresolution images 231 to a display 230 during the interval of theexperiment. In addition, other data 232 can be displayed.

In step 980, the processor 210 determines whether the experiment iscomplete. In some cases, the processor 210 determines whether theexperiment is complete by determining the current time is at the end ofthe duration of the experiment. In other cases, the user of the e-Petrisystem 600 may provide input that determines that the experiment iscomplete. For example, the user may enter a stop experiment command atthe host computer 200. If the experiment is not complete, the nextinterval of the experiment begins and the process goes to step 930. Ifthe experiment is complete, the process ends at step 990.

IV. Subsystems

FIG. 17 is a block diagram of subsystems that may be present in the SPLMsystem 10 or in an e-Petri system 600, according to embodiments of theinvention. For example, the SPLM system 10 and the e-Petri system 600include a processor 410. The processor 410 may include first processor112 and/or second processor 210. The processor 410 may be a component ofthe light detector 160 in some cases. The processor 410 may be acomponent of the illumination source 100 in some cases.

The various components previously described in the Figures may operateusing one or more of the subsystems to facilitate the functionsdescribed herein. Any of the components in the Figures may use anysuitable number of subsystems to facilitate the functions describedherein. Examples of such subsystems and/or components are shown in aFIG. 17. The subsystems shown in FIG. 17 are interconnected via a systembus 425. Additional subsystems such as a printer 430, keyboard 432,fixed disk 434 (or other memory comprising computer readable media),display 436, which is coupled to display adapter 438, and others areshown. The display 436 may include the illuminating display 116 and/orthe image display 230. Peripherals and input/output (I/O) devices, whichcouple to I/O controller 440, can be connected to the computer system byany number of means known in the art, such as serial port 442. Forexample, serial port 442 or external interface 444 can be used toconnect the computer apparatus to a wide area network such as theInternet, a mouse input device, or a scanner. The interconnection viasystem bus allows the processor 410 to communicate with each subsystemand to control the execution of instructions from system memory 446 orthe fixed disk 434, as well as the exchange of information betweensubsystems. The system memory 446 and/or the fixed disk 434 may embody afirst CRM 114 and/or a second CRM 220. Any of these elements may bepresent in the previously described features.

In some embodiments, an output device such as the printer 430 or display436 of the SPLM system 10 or the e-Petri system 600 can output variousforms of data. For example, the SPLM system 10 or the e-Petri system 600can output 2D/3D HR color/monochromatic images, data associated withthese images, or other data associated with analyses performed by theSPLM system 10 or the e-Petri system 600.

It should be understood that the present invention as described abovecan be implemented in the form of control logic using computer softwarein a modular or integrated manner. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will know andappreciate other ways and/or methods to implement the present inventionusing hardware and a combination of hardware and software.

Any of the software components or functions described in thisapplication, may be implemented as software code to be executed by aprocessor using any suitable computer language such as, for example,Java, C++ or Perl using, for example, conventional or object-orientedtechniques. The software code may be stored as a series of instructions,or commands on a CRM, such as a random access memory (RAM), a read onlymemory (ROM), a magnetic medium such as a hard-drive or a floppy disk,or an optical medium such as a CD-ROM. Any such CRM may reside on orwithin a single computational apparatus, and may be present on or withindifferent computational apparatuses within a system or network.

A recitation of “a”, “an” or “the” is intended to mean “one or more”unless specifically indicated to the contrary.

The above description is illustrative and is not restrictive. Manyvariations of the disclosure will become apparent to those skilled inthe art upon review of the disclosure. The scope of the disclosureshould, therefore, be determined not with reference to the abovedescription, but instead should be determined with reference to thepending claims along with their full scope or equivalents.

One or more features from any embodiment may be combined with one ormore features of any other embodiment without departing from the scopeof the disclosure. Further, modifications, additions, or omissions maybe made to any embodiment without departing from the scope of thedisclosure. The components of any embodiment may be integrated orseparated according to particular needs without departing from the scopeof the disclosure.

All patents, patent applications, publications, and descriptionsmentioned above are hereby incorporated by reference in their entiretyfor all purposes. None is admitted to be prior art.

What is claimed is:
 1. An e-Petri dish comprising: a light detector; atransparent layer disposed on the light detector and having a specimensurface; and a processor, wherein the light detector is configured tosample a sequence of sub-pixel shifted projection images of a specimenwhen the specimen is located on the specimen surface and while lightincident the specimen from an illumination source is shifted through aplurality of illumination angles, wherein the illumination sourcecomprises sets of light emitting components at different locations, thesets of light emitting components configured to sequentially illuminateto generate the light from the plurality of illumination angles, whereineach sub-pixel shifted projection image is sampled at a differentsampling time while the illumination source provides light from one ofthe illumination angles, and wherein each sub-pixel shifted projectionimage is associated with light passing through the specimen, and whereinthe processor is configured to generate a sub-pixel resolution image ofthe specimen using image data from the sequence of sub-pixel shiftedprojection images of the specimen and a motion vector, wherein theprocessor is further configured to determine the motion vector, themotion vector associated with a plane of interest and the sub-pixelresolution image is focused at the plane of interest.
 2. The e-Petridish of claim 1, further comprising a well coupled to the transparentlayer, the well configured to contain the specimen.
 3. The e-Petri dishof claim 1, wherein there is no lens in the illumination path betweenthe illumination source and the light detector.
 4. The e-Petri dish ofclaim 1, wherein the illumination source is a coherent light source. 5.The e-Petri dish of claim 1, wherein the light detector and theillumination source are on opposing sides of the specimen surface. 6.The e-Petri dish of claim 1, wherein the illumination angles range from−60 degree to +60 degrees with respect to the sensing surface.
 7. Thee-Petri dish of claim 1, wherein the transparent layer has a thicknessin a range of several hundred nanometers to microns.
 8. The e-Petri dishof claim 1, wherein each of the sets of light emitting components is aset of one or more pixels of a display.
 9. The e-Petri dish of claim 8,wherein the display is a liquid crystal display or a light emittingdiode display.
 10. The e-Petri dish of claim 1, wherein each of the setsof light emitting components is a set of one or more pixels of asmartphone screen.
 11. The e-Petri dish of claim 1, wherein the lightdetector is an image sensor.
 12. The e-Petri dish of claim 1, whereinthe light detector and the transparent layer are in the form of a CMOSimage sensor comprising a PDMS layer disposed on a sensing surface. 13.An e-Petri device comprising: a light detector; a transparent layerdisposed on the light detector and having a specimen surface; anillumination source comprising sets of light emitting components atdifferent locations, the sets of light emitting components configured toprovide light from a plurality of illumination angles to a specimenlocated on the specimen surface; and a processor, wherein the lightdetector is configured to sample a sequence of sub-pixel shiftedprojection images of the specimen when the specimen is located on thespecimen surface and while light incident the specimen from theillumination source is shifted through the plurality of illuminationangles, wherein each of the sub-pixel shifted projection images issampled at a different sampling time while the illumination sourceprovides light from one of the illumination angles, wherein eachsub-pixel shifted projection image is associated with light passingthrough the specimen, and wherein the sequence of sub-pixel shiftedprojection images is sampled at different sampling times corresponds tothe plurality of illumination angles, wherein the processor isconfigured to generate a sub-pixel resolution image of the specimenbased on the sequence of sub-pixel shifted projection images of thespecimen and a motion vector based on the sequence of sub-pixel shiftedprojection images, and wherein the processor is further configured todetermine the motion vector, the motion vector associated with a planeof interest wherein the sub-pixel resolution image is focused at theplane of interest.
 14. The e-Petri device of claim 13, wherein the setsof light emitting components provide light of different lightintensities at different locations.
 15. The e-Petri device of claim 13,wherein the sets of light emitting components provide light of aplurality of wavelengths at different locations; wherein the sequence ofsub-pixel shifted projection images comprises a plurality subsequencesof sub-pixel shifted projection images associated with the plurality ofwavelengths; and wherein the processor is further configured to generatea plurality of sub-pixel resolution images associated with the pluralityof wavelengths and configured to combine the plurality of sub-pixelresolution images associated with the plurality of wavelengths togenerate a multi-color sub-pixel resolution image of the specimen. 16.The e-Petri device of claim 13, further comprising a well defining by aperipheral wall coupled to the specimen surface.
 17. The e-Petri deviceof claim 16, further comprising a cover located over the well.
 18. Thee-Petri device of claim 16, wherein the well includes an array ofchambers.
 19. The e-Petri device of claim 13, further comprising adielectric cage.
 20. The e-Petri device of claim 13, further comprisinga fluid channel.
 21. The e-Petri device of claim 13, wherein the lightdetector includes the processor.
 22. The e-Petri device of claim 13,wherein each of the sets of light emitting components is a set of one ormore pixels of a display.
 23. The e-Petri device of claim 13, whereineach of the sets of light emitting components is a set of one or morepixels of a smartphone screen.
 24. The e-Petri device of claim 13,wherein the light detector is an image sensor.
 25. The e-Petri device ofclaim 13, wherein the light detector and the transparent layer are inthe form of a CMOS image sensor comprising a PDMS layer disposed on asensing surface.
 26. An e-Petri system comprising: one or more e-Petridevices, each e-Petri device comprising a light detector; a transparentlayer disposed on the light detector and having a specimen surface; anillumination source comprises sets of light emitting components atdifferent locations, the sets of light emitting components configured tosequentially shift illumination through a plurality of illuminationangles to a specimen located on the specimen surface; and wherein thelight detector is configured to sample a sequence of sub-pixel shiftedprojection images of the specimen when the specimen is located on thespecimen surface and while light incident the specimen from theillumination source is shifted through the plurality of illuminationangles, wherein each of sub-pixel shifted projection images is sampledat a different sample time while the illumination source provides lightfrom one of the illumination angles, and wherein each sub-pixel shiftedprojection image is associated with light passing through the specimen;and a processor configured to generate a sub-pixel resolution image ofthe specimen based on the sequence of sub-pixel shifted projectionimages from at least one of the one or more e-Petri devices and a motionvector based on the sequence of sub-pixel shifted projection images,wherein the processor is further configured to determine the motionfactor, wherein the motion is associated with a plane of interest andthe sub-pixel resolution image is focused at the plane of interest. 27.The e-Petri system of claim 26, further comprising a relay incommunication with the processor to receive data from the one or moree-Petri devices; and further comprising a host computer in communicationwith the relay to receive data from the one or more e-Petri devices. 28.The e-Petri system of claim 26, wherein the sets of light emittingcomponents provide light of different light intensities at differentlocations.
 29. The e-Petri system of claim 26, wherein the sets of lightemitting components provide light of a plurality of wavelengths atdifferent locations; wherein the sequence of sub-pixel shiftedprojection images comprises a plurality subsequences of sub-pixelshifted projection images associated with the plurality of wavelengths;and wherein the processor is further configured to generate a pluralityof sub-pixel resolution images associated with the plurality ofwavelengths and configured to combine the plurality of sub-pixelresolution images associated with the plurality of wavelengths togenerate a multi-color sub-pixel resolution image of the specimen. 30.The e-Petri system of claim 26, further comprising an incubatorconfigured to maintain a pre-defined environment, the one or moree-Petri devices located within the incubator.
 31. The e-Petri system ofclaim 26, wherein each e-Petri device further comprises a supportholding the illumination source with respect to the specimen surface.32. A method of automatedly generating a sub-pixel resolution image of aspecimen using an e-Petri system at intervals during an experiment, themethod comprising: introducing the specimen on a specimen surface of atransparent layer disposed over a light detector of an e-Petri dish;during each interval, using an illumination source to shift lightincident the specimen through a plurality of illumination angles,wherein the sequentially illuminating different sets of light emittingcomponents of an illuminating display; during each interval, using thelight detector to sample a sequence of sub-pixel shifted projectionimages corresponding to the plurality of illumination angles, whereineach of the sub-pixel shifted projection images is sampled at adifferent sampling time while the illumination source provides lightfrom one of the illumination angles and wherein each sub-pixel shiftedprojection image is associated with light passing through the specimen;and during each interval, constructing a sub-pixel resolution image ofthe specimen based on the sequence of sub-pixel shifted projectionimages and a motion vector; and focusing the sub-pixel resolution imageat a plane of interest by determining the motion vector based on theplane of interest.
 33. The method of claim 32, further comprising duringeach interval, displaying the sub-pixel resolution image of thespecimen.
 34. The method of claim 32, wherein providing light from aplurality of illumination angles comprises sequentially illuminatingdifferent sets of light emitting components of an illuminating display.35. The method of claim 32, wherein each of the sets of light emittingcomponents is a set of one or more pixels of a display.
 36. The methodof claim 32, wherein each of the sets of light emitting components is aset of one or more pixels of a smartphone screen.
 37. The method ofclaim 32, wherein the light detector is an image sensor.
 38. An e-Petridish comprising: an image sensor chip; a transparent well disposed onthe image sensor chip; and a processor, wherein the image sensor chip isconfigured to sample a sequence of sub-pixel shifted projection imagesof a specimen when the specimen is located in the transparent well andwhile light from an illumination source is shifted through a pluralityof illumination angles, wherein the illumination source comprises setsof light emitting components at different locations, the sets of lightemitting components configured to sequentially illuminate to generatethe light from the plurality of illumination angles, wherein eachsub-pixel shifted projection image is sampled at a different samplingtime while the illumination source provides light from one of theillumination angles, and wherein each sub-pixel shifted projection imageis a measurement of light from the illumination source transmittedthrough the specimen, and wherein the processor is configured togenerate a sub-pixel resolution image of the specimen using image datafrom the sequence of sub-pixel shifted projection images of the specimenand a motion vector, wherein the processor is further configured todetermine the motion vector, the motion vector associated with a planeof interest and the sub-pixel resolution image is focused at the planeof interest.
 39. The e-Petri dish of claim 38, wherein each of the setsof light emitting components is a set of one or more pixels of adisplay.
 40. The e-Petri dish of claim 38, wherein each of the sets oflight emitting components is a set of one or more pixels of a smartphonescreen.